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PREFACE 

In  June  1991,  the  Department  of  Defense  of  the  United  States  of 
America  (US  DoD)  and  the  United  Kingdom  Ministry  of  Defence  (UK  MOD) 
agreed  to  merge  their  respective  Ada  compiler  evaluation  test  suites  and 
produce  a  single,  internationally  available  suite. 

The  development  of  the  US  suite,  the  Ada  Compiler  Evaluation 
Capability  (ACEC),  is  managed  by  Mr.  Raymond  Szymanski  of  Wright 
Laboratory  and  was  accomplished  by  Boeing  Defense  and  Space  Group 
Product  Support  Division,  Wichita,  Kansas.  This  effort  is  sponsored  by  the 
Ada  Joint  Program  Office  (AJPO). 

The  development  of  the  UK  suite,  the  Ada  Evaluation  System  (AES)  , 
was  accomplished  by  the  British  Standards  Institute  and  Software 
Sciences  Ltd.  This  effort  was  sponsored  by  the  United  Kingdom  Ministry  of 
Defence. 

The  agreement  between  the  two  governments  established  the  AJPO 
as  the  office  responsible  for  carrying  out  the  merger.  As  a  result,  the 
ACEC  contractor  developed  an  initial  approach  to  merging  the  two  suites 
after  completing  a  substantial  study  of  the  AES.  Their  observations  and 
analysis  were  used  as  a  starting  point  for  Merger  Workshop  discussions. 

The  Workshop  co-chairs  were  Mr.  Raymond  Szymanski  of  Wright 
Laboratory,  Evaluation  &  Validation  Program  Manager,  and  Mr.  Dan  Roy,  of 
the  Software  Engineering  Institute,  Real-time  Embedded  Systems  Testbed 
(REST)  Project  Leader. 

The  accomplishments  of  the  Workshop  are  an  important  contribution 
to  defining  the  merged  product.  Improvements  in  portability,  usability,  and 
completeness  are  expected  as  a  result  of  the  recommendations  made  and 
issues  addressed. 
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1.0  EXECUTIVE  SUMMARY 

1.1  Background 

Dr  John  Solomond,  AJPO  Director,  decided  that  the  Ada  community 
would  be  best  served  by  a  single  Ada  compiler  evaluation  suite.  This  test 
suite  would  embody  the  best  capabilities  of  two  government  developed 
test  suites,  the  ACEC  of  the  US  DoD,  and  the  AES  of  the  UK  MoD. 

In  June  1991,  the  governments  agreed  to  a  suite  merger  with  the  US 
DoD's  AJPO  responsible  for  the  activity.  The  AJPO,  in  turn,  tasked  Mr. 
Raymond  SzymanskI,  the  E&V  Project  manager,  with  management 
responsibility  for  the  merged  product.  Mr.  Szymanski  is  also  responsible 
for  the  Planned  Product  Improvement  cycle  on  the  existing  ACEC. 

As  specified  in  the  merger  agreement,  the  merged  suite  will  have 
unlimited  distribution  to  the  Ada  community  both  in  the  US  and 
internationally. 

1.2  Workshop  Initiation 

The  success  of  the  ACEC,  as  measured  by  over  200  users  and  the 
benefits  derived  from  its  use,  is  attributable  to  several  management 
practices  and  technical  factors  employed  during  its  development.  These 
practices  and  factors  include:  product  peer  review  during  development, 
user  evaluation  between  releases,  selection  of  technically  competent 
developers  and  reviewers,  and  team  membership  consistency.  Additionally, 
participants  from  each  user  sector,  government,  industry  and  academia, 
provided  the  necessary  multiple  perspectives  to  insure  all  user  needs 
were  considered. 

Following  this  successful  formula,  workshop  invitees  included  the 
developers  of  the  ACEC  &  the  AES,  a  variety  of  users  of  both  suites, 
compiler  vendors,  and  independent  evaluators  of  the  ACEC  &  AES.  This  mix 
ensured  that  both  test  suites,  as  well  as  many  types  of  future  merged- 
suite  users,  were  well  represented. 

1.3  Workshop  Philosophy 

The  workshop  was  organized  in  a  fashion  to  permit  the  participants  to  re¬ 
orient  themselves  to  the  details  of  both  test  suites  and  hear  independent 
assessments  of  each.  To  allow  this,  the  developers  of  both  the  ACEC  and 
the  AES  were  invited  to  present  details  of  their  suite's  latest  version,  and 
plans  for  future  versions.  Individuals  who  have  used  either  one  or  both 
suites  were  invited  to  present  their  findings  of  each  suite's  strengths  and 
weaknesses.  Additionally,  the  merger  project  office  was  invited  to 
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present  a  proposed  approach  to  merging  the  two  suites.  These 
presentations  would  then  be  followed  by  nearly  two  days  of  discussion  on 
the  merger  subject. 

1.4  Workshop  Accomplishments 

The  debates  and  discussions  that  ensued  during  the  workshop  were 
vigorous  and  informing.  As  a  result,  the  workshop  succeeded  in  providing 
numerous  positive  recommendations  relevant  to  merging  the  ACEC  and 
AES.  Some  of  those  recommendations  are  listed  below  in  a  non  priority 
order.  Details  of  these  recommendations,  related  issues  and  relevant 
discussion  are  contained  in  the  main  document  sections  and  the 
appendices. 

•  Recommended  "portability",  "usability",  and  "completeness"  as 
primary  requirements  for  the  merged  product. 

•  Recommended  the  merged  product  to  be  an  ACEC  adaptation  of  AES 
technology  and  functionality. 

•  Recommended  a  level  of  technical  review  be  performed  on  AES 
elements  prior  to  inclusion  in  the  merged  suite. 

•  Recommended  the  E&V  Reference  System  as  the  future  home  for  all 
AES  non-compiler-related  assessors. 

•  Recommended  a  priority  for  the  integration  of  AES  compiler- 
related  assessors 

•  Recommended  user  interface  improvements 

•  Recommended  analysis  reporting  improvements 


Accssslon  For 


1  NT:" 
DT  ;  , 
Ih;,-  ,  ; 
Ju.  - 

■  r.l 

□ 

n 

n 

By 

L  i 

’■  :*  ' 

1 

Av.-. 

■  .  ■  1 

iOlat 

K'l 

'  1 

* 

2 


ACEC/AES  MERGER  WORKSHOP 


2.0  ACEC  &  AES  MERGER  WORKSHOP  PROCEEDINGS 

This  section  provides  a  summary  of  the  workshop  presentations, 
merger  issues,  discussions  and  merger  recommendations. 

2.1  Opening  Remarks 

The  ACEC/AES  Merger  Workshop  opened  with  a  welcome  and  an 
introduction  to  the  area  by  Mr.  George  Robertson,  of  Fleet  Combat 
Directional  Systems  Support  Activity  (FCDSSA),  San  Diego.  Mr.  Robertson 
detailed  the  Navy's  commitment  to  the  Ada  Language  and  the  challenges 
they  faced  during  the  upcoming  transition  years.  In  closing,  Mr.  Robertson 
described  FCDSSA' a  usage  of  the  ACEC  test  suite  during  evaluation  of  the 
Navy's  Ada  Language  System  /  Navy  (ALS/N). 

Mr.  Raymond  Szymanski,  Merger  Workshop  co-chair,  welcomed 
everyone  on  behalf  of  Dr.  John  Solomond,  AJPO  Director  and  Mr,  Dan  Roy, 
fellow  co-chairman.  He  stated  the  main  objective  of  the  workshop  was  to 
review  and  refine  an  approach  to  merging  the  ACEC  and  the  AES.  He 
reminded  the  participants  that  they  were  chosen  on  the  bases  of  their 
technical  expertise,  their  ability  to  work  cooperatively  as  part  of  the 
merger  team,  and  for  the  different  professional  perspectives  they  could 
provide.  He  reviewed  the  proposed  agenda  for  the  meeting  and  briefly 
discussed  the  various  presentations  that  would  be  forthcoming. 

Mr.  Dan  Roy  commented  on  the  need  for  the  workshop  attendees  to 
consider  cultural  differences  in  the  development  of  the  two  suites  in 
addition  to  considering  the  views  of  users,  vendors,  and  governments.  He 
stated  that  his  philosophy  for  the  workshop  is  that  the  process  of 
evaluation  is  more  important  than  the  specifics  of  the  technology. 

2.2  Summary  of  Presentations 

The  workshop  was  organized  in  a  fashion  to  permit  the  participants 
to  re-orient  themselves  to  the  details  of  both  test  suites  and  hear 
independent  assessments  of  each.  To  allow  this,  the  developers  of  both  the 
ACEC  and  the  AES  were  invited  to  present  details  of  their  suite's  latest 
version,  and  plans  for  future  versions.  Individuals  who  have  used  either 
one  or  both  suites  were  invited  to  present  their  findings  of  each  suite's 
strengths  and  weaknesses.  Additionally,  the  merger  project  office  was 
invited  to  present  a  proposed  approach  to  merging  the  two  suites.  These 
presentations  are  summarized  below.  The  presentation  vu-folls  can  be 
found  in  the  appendices. 

Note:  Unfortunately,  the  AES  developers  were  unable  to  attend  the 
Workshop.  However,  AES  information  was  presented  by  other  attendees. 
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2.2.1  ACEC  Version  3.0  Product  -  Boeing 

Sam  Ashby,  Kermit  Terrell,  and  Barbara  Decker-Lindsey  of  the 
Boeing  Defense  and  Space  Group  Product  Support  Division  provided  a 
status  report  on  ACEC  Version  3.0  development  and  a  comprehensive 
presentation  on  ACEC  Version  3.0  capabilities. 

Mr,  Ashby  reported  that  ACEC  Version  3.0  testing  has  been  completed 
on  five  target  systems,  including  the  DEC  self-hosted  Ada,  Telesoft  VAX 
self-hosted  Ada,  VAX  hosted  compiler  targeted  to  a  1750A  processor 
(TLD),  Meridian  DEC  Station  self-hosted  (UNIX),  and  Verdix  self-hosted 
Silicon  Graphics.  All  problems  identified  during  testing  were  resolved  and 
delivery  was  made  to  the  customer  on  18  December  1991.  Mr.  Ashby 
concluded  by  stating  that  the  product  is  currently  undergoing  final 
customer  review. 

Mr.  Terrell's  presentation  detailed  the  improvements  made  to  the 
ACEC  in  Version  3.0  which  Included  test  suite  reorganization,  additional 
performance  tests,  new  and  expanded  assessors,  a  new  pre-test 
capability,  and  an  enhanced  user  interface. 

Ms.  Decker-Lindsey's  presentation  detailed  the  capabilities  of  the 
ACEC  Version  3.0  analysis  tools  and  explained  where  Improvements  were 
made  in  analysis  tool  capabilities  and  user  interfaces.  These 
improvements  include  a  user  menuing  system,  an  editable  results  data 
base,  a  data  extraction  tool,  and  a  reduction  in  the  number  of  steps 
required  to  perform  the  analysis. 

2.2.2  AES  Version  2.0  Assessment  -  Boeing 

Mr,  Tom  Leavitt  presented  a  review  of  the  AES  Version  2.0,  based  on 
his  experience  with  running  the  AES,  focusing  specifically  on  the  test 
harness,  specific  performance  tests,  assessors,  and  analysis  and  reporting 
capabilities.  Mr.  Leavitt's  activities  for  this  review  included  reading  the 
doc -mentation,  examining  the  source  code  for  the  performance  tests, 
executing  the  performance  test  groups,  and  running  other  selected  AES 
elements. 

2.2.3  AES  Version  2.0  Assessment  -  SEI 

Mr.  Neal  Altman  presented  a  review  of  the  AES  Version  2,0,  based  on 
his  experience  with  running  the  AES,  focusing  on  the  executable 
benchmark  tests  and  the  test  harness  as  primary  concerns,  with  the 
checklists  and  documentation  as  secondary  concerns.  He  also  discussed 
AES  organizational  issues  and  features. 
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Mr.  Patrick  Donohoe  presented  his  experience  with  ACEC  Version  3.0. 
He  discussed  the  suite's  documentation  and  the  pre-test  setup  steps.  He 
also  discussed  the  performance  tests  that  he  had  run  to  date. 

2.2.5  E&V  Reference  System  Version  3.1  Demonstration  - 

TASC 

Dr.  Bard  Crawford  gave  a  demonstration  of  the  Evaluation  and 
Validation  (E&V)  Reference  System  Version  3.1  which  is  implemented 
with  a  hypertext  capability.  During  this  presentation  he  discussed  the 
purpose  of  the  E&V  Reference  System  and  demonstrated  the  new 
functionality  and  usability  provided  via  hypertext. 

2.2.6  ACEC  &  AES  Merger  Technical  Approach,  -_BQfiing 

Mr.  Kermit  Terrell  presented  a  proposed  approach  to  merging  the 
ACEC  and  AES.  As  background  information  he  provided  a  set  of  high  level 
merged-suite  requirements  along  with  a  list  of  technical  issues  that 
would  need  to  be  addressed  prior  to  the  merger.  Mr.  Terrell  also  presented 
details  of  AES  technology  and  capabilities  which  should  or  should  not  be 
considered  for  inclusion  in  the  merged  product. 

This  presentation  formed  the  basis  for  the  issues,  recommendations, 
and  discussions  that  are  contained  in  the  following  sections.  Therefore, 
these  items  are  not  repeated  in  this  section. 

2.3  Issues  and  Recommendations 

This  section  outlines  the  issues  and  recommendations  that  were 
formulated  during  the  discussion  portion  of  the  Workshop.  The 
participants  were  encouraged  to  raise  issues,  provide  comments  and 
tender  recommendations  as  if  all  issues  could  be  researched  and  all  non¬ 
conflicting  recommendations  could  be  implemented.  This  approach  proved 
successful  in  creating  a  robust  list  of  issues  and  recommendations,  even 
though  the  participants  knew  a  priori  that  the  scope  of  the  merger  could 
not  support  this  assumption. 

The  following  issues  were  raised  at  the  workshon  and  are  briefly 
discussed  in  the  sections  below.  Additional  discussion  on  each  is  provided 
in  the  appendices. 

•  Basic  approach  for  merger 

•  Development  of  a  merger  requirements  document 
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•  Utilization  of  AES  performance  tests 

•  Utilization  of  AES  assessors 


•  Compiler-related 

•  Non-Compiler-related 

•  Functionality  of  user  interface. 

•  Ease  of  test  suite  setup. 

•  Functionality  of  the  reporting  tools. 

•  Functionality  of  the  analysis  tools 


2.3.1  Issue  -  Basic  Merger  Approach 

Should  the  merged  suite  be  an  adaptation  by  the  ACEC  of 
AES  functionality  or  an  adaptation  by  the  AES  of  ACEC 
functionality  ? 

Discussion:  The  basic  approach  to  merging  the  two  suites  should  be 
based  upon  high  level  requirements  that  consider  the  following: 
portability,  usability,  completeness,  ease  of  adaptation,  number  of  users 
of  each  suite,  and  the  types  of  intended  users. 

The  designs  of  the  ACEC  and  AES  were  significantly  influenced  by  the 
developers'  understanding  of  who  the  end  users  were  intended  to  be.  The 
AES,  which  was  designed  for  use  on  a  single  host  by  a  centralized  test 
facility,  is  not  easily  ported  to  new  host/target  combinations  and 
currently  has  few  users.  The  ACEC,  however,  which  was  designed  to  be 
portable  to  accommodate  the  independent  tester,  currently  has  two 
hundred  users  who  test  compilers  on  many  different  host  and  target 
combinations. 

Both  suites  require  an  amount  of  adaptation  by  the  user.  The  ACEC 
was  designed  for  ease  of  adaptation  and  support  is  provided  to  the  user  in 
the  documentation  for  this  process.  The  AES  emphasized  ease  of  use  over 
portability.  For  this  reason  the  adaptation  effort  is  higher. 

A  review  of  the  AES  test  harness  code  reveals  that  it  would  be 
difficult  and  expensive  to  port  this  system  to  hosts  beyond  the  original. 
Porting  this  harness  to  many  hosts  would  not  only  be  cost  prohibitive 
from  a  development  perspective,  but  from  a  maintenance  one  as  well. 
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Another  consideration  is  the  user  base  of  each  suite.  For  the 
hundreds  of  ACEC  users  to  employ  the  AES  method  of  doing  business, 
would  require  a  significant  investment  in  learning  a  new  system.  This  is 
neither  logical  nor  efficient  as  there  are  considerably  more  ACEC  users 
than  AES  users. 

Recommendation;  The  merged  suite  should  be  an  adaptation  by  the 
ACEC  of  AES  technology  and  functionality. 


2.3.2  Issue  -  Development  of  a  Merger  Requirements 
Document 

Should  a  Merger  Requirements  Document  be  produced 
prior  to  initiation  of  the  merger  ? 

Discussion:  The  proposed  approach  to  the  merger  is  a  simple 

blending  of  ACEC  and  AES  technology  and  functionality,  without  the 
addition  of  new  technology  or  functionality.  That  is,  if  the  technology  and 
functionality  does  not  exist  in  either  of  the  suites,  it  will  not  exist  in  the 
merged  suite. 

The  ACEC  and  AES  differ  significantly  in  both  technology  and 
functionality  as  a  result  of  having  development  requirements  that  varied 
significantly.  However,  a  union  of  these  requirements,  brought  about  by  a 
simple  merger,  may  not  allow  the  merged  suite  to  meet  today's  user 
requirements.  The  ACEC,  for  example,  has  not  implemented  each  and  every 
item  on  its  pre-planned  product  Improvement  list.  Items  such  as  compiler 
reliability,  which  neither  suite  has  addressed,  is  a  good  candidate  for 
implementation  in  the  merged  suite.  There  are  many  more  examples  of 
desired  technology  and  functionality  for  the  merged  suite  which  were 
recommended  by  the  workshop  participants. 

Recommendation:  Although  a  formal  requirements  document  for 
the  merged  suite  does  not  need  to  be  developed,  any  established 
requirements  should  not  be  just  a  simple  blending  of  ACEC  and  AES 
requirements.  The  merged  suite  requirements  should  allow  for  technology 
and  functionality  which  currently  does  not  exist  in  either  suite. 


2.3.3  Issue  -  AES  Performance  Tests 

Should  the  AES  performance  tests  be  included  in  the 
merged  suite  ? 
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Discussion:  The  AES  contains  many  tests  which  will  be  useful  in 
the  merged  product  as  they  address  technical  nuances  not  addressed  by 
the  ACEC.  In  areas  that  the  ACEC  does  address,  some  AES  tests  are 
different  enough  to  provide  additional  useful  information.  A  review  of  the 
AES  tests  has  indicated  that  they  should  all  be  thoroughly  reviewed  before 
inclusion  in  the  merged  suite. 

Recommendation:  Incorporate  AES  performance  tests  into  the  merged 
suite  after  a  thorough  review  and  modification  as  necessary.  Where 
appropriate  include  in  existing  ACEC  groups  and  subgroups.  If  necessary, 
create  new  groups  and  subgroups. 


2.3.4  Issue  -  AES  Assessors 

Should  the  AES  assessors  be  included  in  the  merged  suite  ? 

Discussion:  The  assessors  which  are  common  between  the  two 
suites  are  in  the  areas  of  capacity  limits,  diagnostic  messages,  program 
library  manager  and  symbolic  debugger.  These  are  all  compiler-related 
functions.  The  AES  contains  assessors  for  functions  which  the  ACEC  does 
not.  These  assessors  evaluate  both  compiler-related  and  non-compiler- 
related  functions. 

There  is  value  in  providing  the  merged  suite  user  with  additional 
assessors  which  evaluate  compiler-related  functions.  Although  these 
tests  are  not  usually  automatable  they  do  provide  additional  information 
upon  which  to  base  a  selection  decision. 

Recommendation:  Incorporate  AES  compiler-related  assessors  into  the 
merged  suite  after  a  thorough  review  to  eliminate  redundancy  with  ACEC 
assessors  and  perform  modification  as  necessary. 


2. 3. 4.1  Issue,  -  Compiler-related  Assessors 

Which  assessors  from  the  AES  should  be  incorporated  in  the 
merged  suite  and  in  which  priority  ? 

Discussion:  The  following  AES  assessors  were  recommended  for 
inclusion  in  the  merged  suite  and  are  listed  in  priority  order  as 
determined  by  the  workshop  participants. 

•  Profiler 
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•  Cross-referencer 


•  Test  coverage  analysis 

•  Test  bed  generator 

•  Pretty  printing 

•  Stub  generator 

•  Syntax-based  editing 

•  Assertion  checker 

•  Name  expander 

Recommendation:  Include  the  assessors  named  above  in  the  merged 

suite. 


2. 3. 4.2  issue  -  Non-Compiler-related  Assessors 

Should  the  merged  suite  provide  assessors  that  evaluate 
non-compiler-related  tools  ? 

Discussion:  The  AES  assessors  address  both  compiler  and  non¬ 
compiler-related  tools.  The  ACEC  has  restricted  itself  to  compiler- 
related  issues  since  non-compiler  evaluation  issues  were  relegated  to 
the  Evaluation  &  Validation  Reference  System;  like  the  ACEC,  a  product  of 
the  E&V  Project. 

The  increasing  size  of  the  test  suite  is  also  a  concern.  Adding  non- 
compiler-related  assessors  to  the  suite  is  an  unwarranted  and 
unnecessary  growth  when  these  assessors  have  a  natural  home  in  the  E&V 
Reference  System. 

Recommendation:  Incorporate  the  non-compiler-related  assessors 
from  the  AES  into  the  E&V  Reference  System. 


2.3.5  Issue  -  User  Interface 

Should  the  merged  suite  provide  all  current  AES  test 
harness  functionality  ? 
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Discussion:  The  AES  test  harness  provides  the  user  functionality  that  is 
not  provided  by  the  ACEC.  These  functions  include  an  interactively 
accessible  database,  a  command  file  generation  capability,  an  Ada 
program  generation  capability  and  test-harness-level  direct  execution 
mode. 


Inspection  of  the  AES  source  code  for  the  desired  functions  reveals 
that  many  features  are  host  dependent  and,  therefore,  may  not  be  readily 
ported  to  other  hosts.  One  reason  may  be  that  the  operating  system  for  a 
new  host  may  not  be  able  to  provide  the  same  support  to  implement  the 
desired  functions  as  the  original  AES  host  did. 

Recommendation:  Investigate  methods  for  providing  the  desired 
functions  from  the  AES  test  harness  in  a  portable  fashion.  Incorporate 
these  functions  if  they  can  be  implemented  in  a  portable  manner. 

2. 3. 5.1  Issue  ■■  Interactively..  Accessible  Database 

Should  the  merged  suite  provide  an  interactive  capability 
to  determine  the  number  of  tests  that  have  and  have  not  run,  and 
the  tests'  status  ? 

Discussion:  This  capability  would  be  extremely  useful  for  the  user 
who  traditionally  runs  custom  subsets  of  the  tests. 

The  AES  provides  a  capability  to  interactively  determine  which 
tests  have  been  run  and  their  status.  However,  the  data  it  outputs  are 
rather  cryptic  and  sometimes  difficult  to  correctly  interpret.  The  ACEC 
provides  this  information  on  its  test  reports,  after  completion  of  testing. 
Although  the  data  required  is  available  in  the  ACEC,  no  ACEC  mechanism 
exists  to  access  that  data  interactively  during  the  testing  process. 

Recommendation:  Provide  the  merged  suite  user  with  the 
capability  to  interactively  determine  the  number  of  tests  that  have  and 
have  not  run,  and  the  tests'  status.  Provide  this  capability  through 
existing  ACEC  data  structures  if  it  does  not  significantly  expand  the 
database. 


2. 3. 5. 2  Issue  -  Command  File  Generation  Capability 

Should  the  merged  suite  provide  a  command  file  generation 
capability  for  the  purpose  of  implementing  a  highly  interactive 
test  selection  user  interface  ? 
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Discussion:  The  AES  allows  the  user  to  interactively  select 
individual  tests  for  execution  via  a  command  file  generation  capability. 
The  ACEC  allows  the  users  to  select  pre-defined  fixed  groups  of  tests  by 
either  editing  the  existing  command  files  or  creating  new  command  files, 
depending  on  the  execution  host.  However,  many  users  will  be  interested 
in  running  tests  which  may  be  combinations  of  subsets  of  larger  pre¬ 
defined  groupings.  A  powerful,  highly  interactive  test  selection  user 
interface  could  provide  the  functionality  required  by  a  user  who  desires  to 
create  custom  test  groupings. 

The  solution  discussed  may  require  a  database  capability  whose 
development  cost  may  be  far  beyond  the  scope  of  the  merger  effort. 
Additionally,  this  capability  may  not  be  portable  which  is  in  direct 
conflict  with  the  portability  requirement. 

A  compromise  solution  would  be  to  provide  selection  capabilities  on 
pre-defined  subgroups  instead  of  on  individual  tests.  This  approach  may 
alleviate  the  requirement  for  the  database  capability. 

Another  approach  requires  additional  functionality  in  the  report 
generation  tools.  Although  this  does  not  solve  the  selection  problem,  it 
does  produce  only  results  for  desired  tests  by  allowing  the  user  more 
flexibility  in  data  output  selection. 

Recommendation:  The  merged  suite  should  provide  the  capability 
to  select  custom  sets  of  tests  for  execution,  minimally  at  the  ACEC 
subgroup  level.  The  approach  used  must  be  highly  portable  and  as  such, 
shall  avoid  all  non-portable  database  schemes. 


2. 3. 5.3  Issue  -  Ada  Program  Generation  Capabjlitv 

Should  the  merged  suite  provide  an  Ada  program  generation 
capability  to  generate  test  code  7 

Discussion:  The  AES  provides  an  Ada  program  generation  capability  to 
create  test  code.  This  capability  is  useful  when  testing  a  single  system  or 
when  code  is  required  to  determine  compiler  capacity  limits.  However,  the 
most  common  usage  of  the  merged  suite  will  be  to  compare  multiple 
versions  of  a  compiler  or  to  compare  different  compilers.  In  this  mode, 
code  that  is  automatically  generated  may  not  be  at  a  level  of  detail 
capable  of  distinguishing  between  systems.  Also,  there  Is  some  question 
of  repeatability,  i.e.  whether  the  same  code  will  be  generated  each  time 
precisely  the  same  for  each  system  under  test. 
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As  for  code  generation  for  testing  compiler  capacity  limits,  the 
ACEC  already  contains  a  successful  mechanism  for  providing  this 
capability. 

Recommendation:  Automatic  code  generation  capabilities  are  of  limited 
value  except  for  capacity  testing.  This  value  does  not  justify  an 
investment  in  the  the  merger  effort. 

2. 3. 5. 4  Issue  -  Test-Harness-Level  Direct  Execution  Mode 

Shouid  the  merged  suite  provide  a  capability  to  execute 
tests  without  exiting  the  test  harness  ? 

Discussion:  The  AES  provides  an  interactive  mechanism  to  select 
individual  tests  for  execution  from  the  test  harness  level.  The  ACEC  does 
not  provide  this  mechanism. 

The  AES  capability  is  highly  dependent  upon  the  VAX  operating 
system  utility,  STARLET.  This  utility  provides  the  operating  system 
interface  for  the  user.  Although  a  harness  level  test  selection  capability 
is  useful,  it  does  not  provide  enough  utility  to  justify  an  attempt  to 
create  a  portable  capability.  To  begin  with,  the  same  capability  can  be 
produced  in  a  portable  fashion  which  simply  requires  the  user  to 
temporarily  exit  the  harness  to  execute  the  necessary  command  files. 
Second,  no  assumptions  can  be  made  about  the  availability  of  a  STARLET- 
llke  utility  on  any  other  operating  systems.  Therefore,  if  they  did  not 
exist  they  would  have  to  be  created  by  ACEC  users  for  each  compilation 
system,  significantly  increasing  the  effort  required  to  adapt  the  test 
suite. 

Recommendation:  Do  not  implement  a  harness-level  direct  execution 
mode  in  the  merged  suite. 


2.3.6  Issue  -  Test  Suite  Setup 

Should  ease  of  setup  be  a  primary  requirement  ? 

Discussion:  As  the  merged  suite  will  be  portable  and  designed  to 
accommodate  the  individual  user,  and  not  a  large  government-run  test 
facility,  great  care  must  be  taken  in  developing  an  appropriate  set-up 
process.  Consideration  must  be  given  to  the  fact  that  the  user's  host  and 
target  combinations  are  numerous  as  will  be  their  experience  level  in 
using  compiler  evaluation  suites.  It  is  therefore  essential  that  the  user  be 
provided  with  considerable  written  assistance  for  the  purpose  of 
initiating  the  evaluation  process.  Although  this  initiation  requires  the 
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accomplishment  of  a  limited  number  of  steps  prior  to  actually  executing 
the  test  suite,  if  they  are  not  accomplished  then  the  suite  cannot  be 
successfully  executed. 

An  acceptable  set-up  process  should  meet  the  following 
requirements: 

•  Consists  of  quality  documentation 

•  Enumerates  the  depth  of  knowledge  required  by  the  tester. 

•  Identifies  key  milestones. 

•  Completely  and  unambiguously  defines  the  setup  process  and 

procedures. 

•  Considers  the  variability  of  compilers. 

•  Provides  a  logical  approach  to  the  problem. 

Recommendation:  Use  the  ACEC  set-up  process  as  a  framework  for 
developing  the  merged  suite  set-up  process  and  ensure  that  it  meets  the 
requirements  listed  above. 


2.3.7  Issue  -  Report  Styles 

Should  the  analysis  reports  favor  the  management-level 
reader  or  the  evaluation  expert-level  reader  ? 

Discussion:  The  AES  produces  one  type  of  report.  This  report  is 
used  to  document  results  for  a  single  system  and  is  aimed  at  the 
management-level  reader.  The  ACEC  produces  two  types  of  reports.  One  is 
used  to  document  strengths  and  weaknesses  in  a  single  system,  while  the 
other  is  used  to  compare  results  from  multiple  systems.  Both  are  aimed  at 
the  compiler  evaluator-level  reader. 

The  advantage  of  a  management  level  report  is  that  conclusions  are 
drawn  for  the  reader  who  does  not  have  to  do  any  analysis.  The 
disadvantage  is  the  reader  is  given  little  if  any  opportunity  to  question 
the  conclusions,  examine  the  results,  and  draw  one's  own  conclusions.  The 
evaluator-level  reports  provide  significant  amounts  of  data  and  require 
the  reader  to  understand  the  technical  issues  and  the  process  involved  in 
reaching  conclusions. 

There  is  a  need  for  both  types  of  reports,  one  for  management  and 
one  for  the  evaluator.  As  requests  for  evaluation  results  by  procuring 
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agencies  become  more  commonplace,  the  need  for  the  former  type  of 
document  will  increase  accordingly.  These  agencies  are  not  expected  to 
retain  compiler  evaluation  experts  who  are  capable  of  interpreting 
evaluator-level  reports.  As  the  number  of  new  compilers  continues  to 
increase  the  need  for  an  evaluator-level  report  will  increase  also.  The 
anticipated  increase  in  the  number  of  new  compilers  is  a  result  of 
anticipated  changes  to  the  Ada  language  via  the  Ada9X  language  revision 
project. 

Recommendation:  Provide  a  configurable  analysis  report 

capability  which  selectively  provides  for  the  needs  of  both  management 
personnel  and  the  compiler  evaluators. 


2.3.8  Issue  -  Assessor  Report  Capabilities 

Should  the  merged  suite  provide  the  capabiiity  to  perform 
comparative  anaiysis  of  assessors  ? 

Discussion:  Any  quantification  of  data  will  provide  a  management 
level  reader  with  an  analysis  report  they  usually  seek  to  avoid  doing 
personally.  On  the  other  hand,  where  the  quantified  data  is  qualitative  in 
nature,  the  reader  will  be  done  a  disservice  in  drawings  conclusions  from 
this  data. 

Since  the  ACEC  provides  a  comparative  analysis  capability  for  the 
performance  tests,  many  users  will  expect  the  merged  suite  to  provide  the 
same  type  of  capabilities  for  the  assessor  results.  As  a  minimum,  results 
from  each  system  should  be  output  next  to  each  other  in  a  columnar 
fashion  to  permit  easy,  manual  comparison  of  the  results 

Recommendation:  Investigate  comparative  analysis  of  assessor 
results  for  the  merged  suite  to  determine  the  utility  of  this  capability.  If 
it  proves  worthwhile,  implement  this  capability  in  the  merged  suite. 
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Ada  Compiler  Evaluation  Capability  (ACEC) 


Contract  Number 
F3361 5-86-01 059 
WL/AAAF 


Boeing  Defense  and  Space  Group 
Product  Support  Division 


Where  we  are 


Version  3.0  Testing  completed 
Ran  on  5  systems 

identified  and  resolved  205  problems 
Version  3.0  to  the  customer  18  Dec  91 
Tapes  (3134  files  -  24  megabytes) 
Documentation  (printed  and  on-line) 
Under  final  customer  review 
Maintenance 


ffOtiNG 
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r 

A 

Documentation 

User's  Guide  (352  pages) 

How  to  set-up  and  run 
Reader's  Guide  (189  pages) 

Why  and  how  to  interpret 
Version  Description  Ooc  (398  pages) 
Descriptions  and  references 


worlittiep 
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Overview  of  Release  3.0 
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EXECUTION-TIME  TEST  GROUPS 


Group  Nam*  Ai 


Application 
Arlthmatle 
Claaaical 
Data  Storage 
Data  Strueturaa 
Dalaya  and  Timing 
Exception  Handling 
Ganarica 

Input/Output  (MIS) 

Mwealianaoua 

Optimizatlona 

Program  Organization 

Statamanta 

Storage  Reclamation 

Subprograms 

Systematic  Compile  Speed 

Tasking 


Abbrev  Number  of  subgroups 


Worn  Shop 

BOBNO 

1/1M2  7 

r 

ASSESSOR  GROUPS 

Group  Name  Abbrev 

Subgroup  Name 

Abbrev 

Capacity  Assessor 

yc_ 

compiiejime 

ct 

runjime 

rt 

Debugger  Assessor 

yb_ 

Diagnostics  Assessor 

y<*_ 

compiler_errors 

ce 

compiler  warnings  cw 

link  time 

It 

run_tlme 

rt 

Library  Assessor 

yL 

y 

WwkHiep 
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SUPPORT  GROUPS 


Group  Name 

Abbrev 

Subgroup  Name  Abbrev 

Analysis 

2a_ 

Comparative  Analysis 

ca 

Condense 

cn 

Menus 

mn 

Single  System  Analysis 

sa 

Command  Files 

2C_ 

Documentation 

zd 

Global  &  Timing  Files  zg_ 

Math  Packages 

zm 

Pretest 

2P_ 

W0rk  Shop  BOEiNO 


COMMAND  FILES 


Organized  by  group  &  subgroup 
Standardized  command  spelling 
Glossary  of  commands 
Ada  programs  for  difficuit>to-adapt  steps 
Compilation  time  stamps,  calculations 
Capacity  test  calculations 


Worti  strap 


BOBMO 


mvn  10 


ACEC/AES  MERGER  WORKSHOP 


Performance  Tests 


Execution  Time  1627  tests 

Code  Size  1627  tests 

Compile  Speed  588  tests 

Link  Speed  571  tests 

Compile  and  Link  Speed  588  tests 

Systematic  Compile  Speed  Group  92  tests 


Wam  Shop 
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SYSTEMATIC  COMPILE  SPEED 


compilation_unit_slze 

compile_time_arithiTietic 

generics 

optimization 

pragmajniine 

programJibrary_size 

smart_recompilation 

source_presentation 

subunits 

symboi_table_size 
with  ciauses 


liumtZfiC 

45 

2 

12 

6 

3 
2 
6 

4 
6 


NEW  PERFORMANCE  TESTS 


Test  Area 

Array  of  records  vs  record  of  arrays  vs  parallel  arrays 
Zero  vs  non*zoro  based  arrays 
Coding  style:  CASE  vs  IF 
Coding  style:  exception  raising  vs  explicit  IF 
Reclamation  tost  using  function  returning  an 
unconstrained  type  In  several  contexts 
Algebraic  simplification  "handedness"  bias 
Allocate  statically  sized  storage  In  blocks 
UNCHECKEO_CONVERSION  between  arrays  and  records 
Passing  Integer  parameters 
’•*”  and  functions  for  TIME  and  DURATION 
Reordering  expressions 
High-precislon  temporaries 


WwkatWf  BOBNQ 


Number 
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NEW  PERFORMANCE  TESTS 
CONTINUED 


T«8t  Araa 

Hutnbtc 

SELECT  whh  variable  number  of  ACCEPT  altamathraa 

4 

Algorithm  uaad  In  aalactiva  wait 

2 

Order  of  evaluation  of  guarda  in  a  aelective  wait 

1 

Variability  of  exception  proeeaaing  time  with  number  of  taaka 

2 

Reclamation  teat  for  taak  created  via  allocation 

1 

Variability  of  taak  creation  with  a  pre-exiating  active  taaka 

2 

Taak-awitch  time 

3 

Scheduling  of  taak  or  maatar  on  creation 

1 

Taak  aeheduling  after  Interrupt 

1 

A  rule^aed  expert  ayatem 

1 

Caching/paging 

36 

Pipelining 

8 

Shared  variable 

2 

V 

RUN-TIME  MEMORY  SIZE 


Determine  size  of  run-time  objects 
Write  as  performance  tests  with  ancillary  data 
Use  'SIZE  where  possible,  otherwise  'ADDRESS 
Variability  with  respect  to  common  optimizations 
Structures  to  measure 
Task  control  blocks 
Activation  records 
Variant  records 

Objects  of  an  unconstrained  type 


Wartitliep 
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Overview  of  Release 


3.0 


structure 

Performance  Tests 


sjiMitS 


Pretest 


Gathering  Data 
Analysis  Tools 
Documentation 


Work  Shop  BOEINa 


ASSESSORS 


Diagnostic  Assessor 
Debugger  Assessor 
Library  Assessor 
Capacity  Assessor  -  New 


For  each  assessor 

Readme  file 
Report  template 


WwliShsp 


1/IM2  1* 
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DIAGNOSTIC  ASSESSOR 


C-IA 


ACEC/AES  MERGER  WORKSHOP 


DEBUGGER  ASSESSOR 


29  debugging  scenarios 

User  performs  debugging  operations 

New: 

Labeis 


Line  numbered  flies 
Tests: 

Functional  capabilities 

Performance 

Capacity 


WorhfHiop 


boom 


LIBRARY  ASSESSOR 


22  scenarios 
New: 


Times,  sizes  collected  in  Systematic  Compile 
Speed  group 
Tests: 

Functional  capabilities 

Performance 

Capacity 


Work  Shop 


BOEtm 
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COMPILE-TIME  TESTS 


Source  code  generated  at  time  of  test  by  supplied 
source  generators 


Tests  static  limits  definable  at  compile  time: 
Quantity  -  names,  tasks,  elements,  etc. 

Size  -  literal  pool,  declarative  region 
Depth  of  nesting  -  iF,  generics,  subunits,  etc. 


WOfkShop 
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RUN-TIME  TESTS 


May  result  in  system  crash  on  weaker  systems 

Tests  dynamic  features  defined  at  run  time: 
Qu'  ntity  -  tasks,  objects,  elements,  etc. 

Size  -  arrays,  collection,  data  segment 
Depth  of  nesting  -  subprogram  calling 


[ 

Overview  of  Release  3.0 

structure 

Performance  Tests 

Assessors. 

OLenceisiti 


Gathering  Data 
Analysis  Tools 
Documentation 
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Purpose 

Aid  user  in  getting  started 
Organize  adaptation  effort 
Provide  usefui  system  information 
Prepare  to  execute  test  suite,  anaiyze  data 
Contents 

Readme  fiie  -  zp_rdmel.txt 
Test  programs,  command  files 
Report  template 


PRETEST 

CONTINUED 


1. 

Access  Ada  Compilation  System 

2. 

Test  label'AOORESS 

3-4. 

System  Clock/Calendar  Tests 

5. 

Compile  Baseline  Flies 

{Mandatory} 

6. 

Test  Inner  Timing  Loop  Iteration  Count 

7-9, 

Test  Math  Package  Adaptation 

10. 

Test  Preprocessor  zgjncid 

11. 

Test  Performance  Command  Files 

(Mandatory) 

12. 

Compile  Analysis  Tools 

13. 

Test  Condense 

14. 

Test  Comparative  Analysis 

^ _ 

_ y 

Work  Shop 
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Overview  of  Release  3.0 


structure 

Performance  Tests 

Assessors 

Pretest 

Analysis  Tools 
Documentation 

V 

Work  SItop  BOEINO 


COLLECTION  OF  RESULTS 


Test  results  written  to  standard  output 
Compilation  log  (host) 

Execution  log  (target) 

Save  logs  to  text  files 
Input  to  analysis  phase 


_ 

Shop  BOBNQ 
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EXECUTION  LOG 


woffesriop 
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CONDENSE 


Call  from  Menu,  SSA,  CA,  or  batch 
Convert  log  files  to  database  files 
Run-time  error  diagnosis 
Compute  compilation  times 
Cross  check  execution,  compilation  results 
Incremental  mode  adds  to  database 
Optional  reports 
No  Data  Report 
Exceptional  Data  Report 
Multiple  Data  Report 


WerkSfwp 
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Execution  time/Code  size 
Compilation  time/Link  time 
Text  files 

Readable,  modifiable  by  user 
By  group,  subgroup 
Duplicate  results  adjacent 
One  result  selected  for  analysis 
Input  to  Comparative,  Single  System  Analysis 


Comparative  Analysis 


Groups 

Summary  of  ail  groups 
Application  profile  mode 
System  factors  &  confidence  intervals 
Outliers 
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r 

Main  Menu 

U^Mii 

a.  -CONDENSE 

b.  -COMPARATIVE.ANALYSIS 

c.  -SINGLE  SYSTEM_ANALYSIS 

HElp  QUK  NExt 

Salact  1  tool  (saparato  seiaetions  with  comma): 

=>  "b,na"  <cr> 

) 

Work  SHOO  BOEtNO 


Workshop 


BOBNa 


^nvn  40 


C-24 


ACEC/AES  MERGER  WORKSHOP 


Metrics  Menu 


a.  -EXECUnON.TIME 

:s  .tim 

b.  -CODE.SIZE 

:=.slz 

c.  -COMPILATION.TTME 

:=  xmp 

d.  -UNK.TIME 

:=  .Ink 

e.  <COMBINED  COMPILATION_UNK_T1ME 

:=  .cml 

f.  -All  Metrics 

HElp  Quit  MAIn 

NExt  PRevlous  OEfauit  names 

Select  1  or  more  metrics  (separate  selections  with  comma): 
=>  ”a,c,ne"  <cr> 


^  Groups  Menu 

«...  o  . 

a.  -APPUCATION 

ssppllcoo 

b.  -AArTHMETIC 

sartthmOO 

e.  -CLASSICAL 

sdassWO 

d.  -OATA.STOBAQE 

xsteragOO 

•.-DATA  STRUCTURES 

sslructOO 

(.  -OEUW_ANO_TTM(NQ 

xdslaysOO 

g. -EXCEPTION  HANOUNQ 

X  sietplOO 

h.  -GENERICS 

xqsnsflOO 

1.  -INPUT  OUTPUT 

X  Input  00 

).  -MISCELLANEOUS 

X  mIseslOO 

k.  -OPTIMIZATIONS 

X  opUmWO 

I.  -PROGRAM  ORGANIZATION 

xprograOO 

m.-STATEMENTS 

xstatsmOO 

n. -STORAGE  RECLAMATION 

X  raclamOO 

0.  -SUBPROGRAMS 

xsubproOO 

p.  -SYSTEMATIC  COMPILE  SPEED 

X  tystsmOO 

q.  -TASKING 

X  taaUnOO 

^  r.  -All  Groups 

y 

ACEC/AES  MERGER  WORKSHOP 
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CA  Report  Options 


- — —  CA  Report  ^tions - 

a. -SUMMARY_REPORT 

b.  -FULL_REPORT  and  SUMMARY_REPORT 

c.  -SUMMARY_OF_ALL_GROUPS_REPORT 

d.  'All  Above 

:=  summryOO 

•.-SPECIAL.REPORT 

:s  speclaOO 

f.  'Write  all  reports  In  current  request  to  one  file 

:=  comparOO.rpt 

g.  -Change  length  of  output  line  to 

:s80 

QUK  MAIn 
PRttvIoua  DEfault  namss 


Salaet  1  or  mora  reports  (separata  selections  with  comma): 
^=>  "b.t.ns"  <cr> 


Run  or  Save  Request 


Current  Selection  Is 

PRCX3RAM  :  COMPARATIVE  ANALYSIS 
SYSTEMS  :  systemj 
system_2 
system's 

METRICS  :  EXECUTION  TIME,  COMPILAT10N_TlME 
GROUPS  :  DATA.STORAGE,  STATEMENTS,  TASKING 
OPTIONS  :  SUMMARY_REPORT,  FULL.REPORT, 

One  output  file. 

Output  line  length:  80 

a.  -Run  Immediately 

b.  •Store  request  In  new  Request  file 

c.  'Append  request  to  existing  Request  file 


HElp  Quit  MAIn  PRavlous  DO  request 

Select  1  option,  and  enter  "DO"  to  apply  (separate  selections  with  comma): 

=>"a,do”<cr> 

1.^ _  ^ 


Wofkttiep  BOGNO  44 
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USER'S  GUIDE 


Pretest  directions 

Readme  files 

Trouble  shooting  guide 

Glossary  of  commands 

Steps  for  adding  tests 

Non-generic  version  of  MATH 

Running  on  a  simulator 

Groups  and  subsets  of  tests 

Changing  compilation  options 

Modifying  tests  to  use  system-dependent  features 


READER'S  GUIDE 


Organization  of  the  test  suite 
Citations  to  other  works 
Report  reviews 

Interpretation  the  analysis  reports 
interpreting  tests  with  system-dependent  modifications 
Interpretation  of  compilation  time  results 
Implementation  trade-offs 


ACEC/AES  MERGER  WORKSHOP 


VERSION  DESCRIPTION 
DOCUMENT 


Test  problem  descriptions 
Test  problem  to  source  file  map 
Tape  description 
ACEC  keyword  indexes 
Quarantined  test  problems 
Mapping  of  old  to  new  names 
System  dependent  test  problems 
Optimization  techniques 
Assessor  information 


Work  SIMP 


BOGNQ 


inewi  sp 
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Pretest 


Include 


Report  Report  Report  Report 

Form  Form  Form  Form 


Report  Report  Report 


Overview  of  the  ACEC  Version  3 


c-.in 
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Code  Size  Report 


Code  Size  Report  :  (physical  lines) 


—  average  bytes  per  lines  :  10.69 

—  based  on  total  line  count  of  20160 


—  HIGHEST  —  Test  :  io_tx_io_09 

—  bytes  per  line  :  757.00  —  line  count  :  1 

—  bytes  per  semicolon:  757.00  —  semicolon  count:  1 

—  LOWEST  —  Test  :  sr  ex_explicit_04 

—  bytes  per  line  ;  O.OS^  —  line  count  :  84 

—  bytes  per  semicolon:  0.17  —  semicolon  count:  29 


Code  Size  Report  :  (semicolons) 


—  average  bytes  per  semicolons  :  16.23 

—  based  on  total  semicolon  count  of  13281 


—  HIGHEST  —  Test  :  io_tx_io_09 

—  bytes  per  line  :  757.00  —  line  count  :  1 

—  bytes  per  semicolon:  757.00  —  semicolon  count;  1 


—  LOWEST  —  Test  :  po  pa_d_library_05 

—  bytes  per  line  :  O.ltT  ”  —  line  count  :  48 

—  bytes  per  semicolon:  0.14  —  semicolon  count'  35 


Code  Size  Report  :  (Examples) 


—  Test  :  cX 

—  bytes  per  line  .  10.32  —  Line  count  47 

—  bytes  per  semicoxon:  19.40  —  semicolon  coan.t  2-5 


— 

—  Test  : 
bytes  per 
bytes  per 

cl_dh_dhrys_ 
line  : 

semicolon: 

02 

7.00 

13.16 

—  line  count  : 

—  semicolon  count: 

47 

25 

—  Test  : 

cl  dh  dhrys 

03 

bytes  per 

line  : 

7.00 

—  line  count 

47 

— 

bytes  per 

semicolon: 

13.16 

—  semicolon  count: 

25 

—  Test  : 

cl  wh  whet  01 

bytes  per 

line  : 

29.13 

—  line  count  : 

134 

— 

bytes  per 

semicolon: 

44.36 

—  semicolon  count: 

88 

v_91 
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Compile  Speed  Report 


Compilation  Speed  :  (physical  lines) 


-  average  lines  per  minute  ;  111.12 

—  based  on  total  line  count  of  211908.0 

- HIGHEST  —  File  ;  ap  )cfm01 

—  lines  per  minute  ;  1570.75 

—  semicolons  per  minute:  811.76 

line  count  : 

semicolon  count: 

4762 

2461 

-  LOWEST  —  File  :  sy_cum21 

—  lines  per  minute  :  0.90 

—  semicolons  per  minute:  0.69 

line  count  : 

semicolon  count: 

209 

162 

Compilation  Speed  :  (semicolons) 


—  average  semicolons  per  minute  :  66.66 

—  based  on  total  semicolon  count  of  127120.0 


—  HIGHEST  —  File  :  po  msm09 

—  lines  per  minute  :  379. T5  —  line  count  :  4348 

—  semicolons  per  minute;  1055.26  —  semicolon  count;  12095 


—  LOWEST  —  File  ;  sy_cum21 

—  lines  per  minute  :  0.90  —  line  count  :  209 

—  semicolons  per  minute;  0.69  —  semicolon  count:  162 


Compilation  Speed  :  (Examples) 

—  File  :  cl  dhmOl 

—  lines  per  minute  : 

—  semicolons  per  minute: 

578.57 

280.87 

—  line  count  : 

—  semicolon  count; 

756 

367 

—  File  :  cl_dhm02 

—  lines  per  minute  : 

—  semicolons  per  minute: 

615.72 

306.21 

—  line  count  ; 

—  semicolon  count: 

744 

370 

—  File  :  cl_dhm03 

—  lines  per  minute  : 

—  semicolons  per  minute: 

619.89 

309.12 

—  line  count  : 

—  semicolon  count: 

748 

373 

—  File  :  cl_whm01 

—  lines  per  minute  ; 

—  semicolons  per  minute: 

306.48 

173.15 

—  line  count  : 

—  semicolon  count: 

331 

187 

V  91 
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Failure  Analysis  Report  -  Execution  Results 


Failure  Analysis  :  by  Group  and  by  Type  of  Failure 


Groups 

1 

Data  Summary 

Categories 

Valid  CmpT  RunT  noDa  Depn  Pkng  Unrl  XcsT  Dely  Vrfy  Othr 

Total 

application 

1  73 

13 

0 

0 

0 

0 

0 

0 

0 

0 

0 

86 

arithmetic 

1  108 

0 

0 

0 

0 

0 

4 

0 

0 

0 

0 

112 

classical 

1  83 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

83 

data  storaq 

1  91 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

91 

data  struct 

1  224 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

225 

delays  and 

1  26 

0 

0 

0 

0 

0 

0 

0 

8 

7 

0 

41 

exception  K 

1  58 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

58 

generics 

1  24 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

24 

input  outpu 

1  105 

0 

0 

0 

0 

0 

3 

0 

0 

0 

0 

108 

misceTlaneo 

1  17 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

17 

optimizatio 

1  304 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

305 

program  org 

1  74 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

74 

statements 

1  80 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

80 

storage  rec 

1  47 

0 

0 

0 

0 

0 

0 

13 

0 

0 

0 

60 

subprograms 

1  79 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

79 

systematic 

1  75 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

75 

tasking 

1  98 

1 

0 

0 

1 

0 

1 

0 

8 

0 

0 

109 

Totals 

1566 

14 

1 

0 

1 

0 

9 

13 

16 

7 

0 

162^ 

Failure  Analysis  Report  ~  Co"ipilat  ic"  Results 


Failure  Analysis  :  by  Group  and  by  T-.  .  of  Failure 


Groups 

1 

Data 

Summary 

Categories 

Valid 

CmpT 

RunT 

noDa 

Depn 

Pkng  Unrl 

XcsT 

Dely 

Vrfy 

othr 

Total 

application 

1 

35 

0 

0 

2 

0 

0 

0 

0 

0 

0 

0  1 

37 

arithmetic 

1 

22 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  i 

22 

classical 

1 

38 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  1 

38 

data  storaq 

1 

14 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  1 

14 

data  struct 

1 

50 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  1 

50 

delays  and 

1 

10 

0 

0 

2 

0 

0 

0 

0 

0 

0 

0  1 

12 

exception  H 

1 

17 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  1 

17 

generics 

1 

5 

0 

0 

0 

0 

0 

0 

n 

0 

0 

0  1 

5 

input  outpu 

1 

22 

0 

0 

1 

0 

0 

0 

6 

0 

0 

0  1 

23 

misceTlaneo 

1 

7 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  1 

7 

optimizatio 

1 

64 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0  1 

65 

program  orq 

1 

20 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  1 

20 

statements 

1 

17 

0 

0 

0 

0 

0 

0 

0 

0 

0 

8  1 

17 

storage  rec 

1 

51 

0 

0 

0 

0 

0 

0 

0 

0 

0 

8  1 

51 

subprograms 

1 

16 

0 

0 

0 

0 

0 

0 

0 

0 

0 

8  1 

16 

systematic 

1 

91 

0 

0 

1 

0 

0 

0 

0 

0 

0 

8  i 

92 

tasking 

1 

100 

0 

0 

2 

0 

0 

0 

0 

0 

0 

8  1 

102 

Totals 

579 

0 

0 

9 

0 

0 

0 

0 

0 

0 

8  1 

588 
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Ancillary  Data 


Ancillary  Data  -  List 


>>>  ap_kf_kalman 

>>>  approximate  time  per  filter  call:  4088.5  number  of  iterations:  800 


>>>  cl_ac_acker  01 

>>>  time  per  call  is  12.0 


>>>  cl_ac_acker  02 

>>>  time  per  call  is  17.9 


>>>  cl_dp_task_02 

>>>  Time  per  rendezvous  -  582.1 


>>>  cl_dp_task_03 

>>>  time  per  rendezvous  -  606.6 


>>>  cl_dp_task_04 

>>>  time  per  rendezvous  -  581.1 


>>>  cl_dp_task_05 

>>>  time  per  rendezvous  -  449.1 


>>>  cl_db_dhrys_01 

>>>  Dhrystones  per  second,  checking:  2553.99 


>>>  cl_dh_dhrys_02 

>>>  Dhrystones  per  second,  checking  suppressed:  3975.81 


>>>  cl_dh_dhrys_03 

>>>  Dhrystones  per  second,  optimize( space ), nochecking:  3960.90 


>>>  dr_ao_ar ray_oper_32 

>>>  2-D  arrays  allocated  in  row-major  order. 


>>>  dr_ss_tcb_01 

>>>  Task  Control  Block  size  is  256  bits,  32  8-bit  bytes 


>>>  DR_SS_ACTIV_REC_01 

>>>  activation  record  size  is  416  bits,  52  8-bit  bytes 


>>>  DR_SS_ACTIV_REC_02 

>>>  activation  record  size  is  352  bits,  44  8-bit  bytes 
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Dead 

Code  Elimination 

Description 

Optimized? 

Time 

:  op  de_dead  05  (  6.6  ] 

ar_io_integer_oper_01  { 

1  vs 

0.4  ) 

no 

Size 

:  op  de_dead  05  (  144.0  i 

ar  io_integer_oper  01  ( 

1  vs 

32.0  ) 

op_de_dead_05 

FOR  i  IN  int' ( 1 ) . .int' ( 5 ) 

LOOP  ii  :■  i  ; 

END  LOOP  ; 
ii  0  ; 

—  dead  assignments  within  loop,  killed  by  assignment 

—  after  exit. 


ar_io_integer_oper_01  kk  1  ; 


Dead 

Code  Elimination 

Description 

Optimized? 

Time 

:  op_de_dead_06  ( 
st_nu_null_01  ( 

1.0 

0.0 

)  vs 
) 

yes 

Size 

:  op_de_dead_06  ( 
st_nu_null~01  ( 

48.0 

0.0 

)  vs 
) 

op_de_dead_06 

DECLARE 

xyz  :  real  ; 

BEGIN 

xyz  ;=  yy  *  zz  ; 

END  ; 

—  dead  assignments  within  a  block.  Variable  assigned  to 

—  local  which  is  not  referenced  before  block  is  exited. 


St  nu  null  01  NULL  ; 
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Optimizations 


Dead 

Code  Elimination 

Description 

Optimized? 

Time 

:  op_de  dead  02  ( 

ar_io_integer_ope  r_04 

1.0  )  vs 
{  0.3  ) 

maybe 

Size 

:  op__de_dead  02  ( 

ar  io""integer  oper  04 

80.0  )  vs 
(  40.0  ) 

op_de_dead_02  ii  :*  11  ;  ii  :»  mm  ; 

—  first  assignment  is  dead 

—  Optimization  test  for  dead  assignment 

elimination  on  integers. 

ar_io_integer_oper_04 

kk  :«  11  ; 

Dead 

Code  Elimination 

Description 

Optimized? 

Time 

:  op_de_dead_03  ( 
ar_f l_f lt_oper_02  ( 

1.9  )  vs 

1.0  ) 

maybe 

Size 

;  op_de__dead_03  ( 
ar_f l“f lt_oper_02  ( 

96.0  )  vs 

48.0  ) 

op_de_dead_03  xx  :•  yy  ;  xx  :«  zz  ; 

—  first  assignment  is  dead 

—  dead  assignment  elimination;  floating  point  variable 


ar_f l_f lt_oper_02  xx  yy  ; 


Dead 

Code  Elimination 

Description 

Optimized? 

Time 

:  op_de_dead_04  ( 
st_nu_null^01  ( 

0.0 

0.0 

)  vs 
) 

yes 

Size 

:  op_de_dead_04  ( 
st_nu_null_01  ( 

0.0 

0.0 

)  vs 
) 

op_de_dead  04  xx  xx  ; 

—  Assign  Tloat  variable  to 

itself . 

st_nu_null_01  NULL 

7 
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Language  Feature  Overhead 


Subprogram  Calls:  With  0..3 

Parameters 

Test 

Execution 

Bar 

Similar 

Name 

Time 

Chart 

Groups 

su  se  external  01 

6.7 

it  it  it  ii  it  it  ^  | 

1 

su  se  external  02 

8.8 

********* 

1 

su  se  external  03 

14.9 

********★★**★  * 

1 

su_se_external_04 

23.3 

************************ 

1 

Code  Size 

su  se  external  01 

48.0 

*  *  * 

su  "se  external  "02 

200.0 

*********** 

su  "se  external~03 

328.0 

***************** 

su_se  external_04 

456.0 

************************ 

Individual  Test  Descriptions 


su_se_external_01  procO  ; 

—  simple  procedure  with  no  parameters;  call  to  library  scope 

—  procedure  :  body  is  null. 


su_se_external__02  prod  (  xx  )  ; 

—  simple  procedure  with  one  IN  OUT  floating  point  parameter, 

—  declared  in  external  library  unit  :  body  is  null. 


su_se_extecnal_03  proc2  (  xx  ,  yy  )  ; 

—  simple  procedure  with  two  IN  OUT  floating  point  parameters, 

—  declared  in  external  library  unit  :  body  is  null. 


su_se_external_04  proc3  (  xx  ,  yy  ,  zz  )  ; 

— "simple  procedure  with  three  IN  OUT  floating  point  parameters, 
—  declared  in  external  library  unit  :  body  is  null. 
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Language  Feature  Overhead 


Performance 

Range 

Significant 

Difference? 

Missing 

Tests 

Total 

Tests 

Page 

Subprogram  Calls: 

With  0..3  Parameters 

54 

6.7  ..  23.3  yes  0  4 


Optimizations 


Yes 

Maybe  No 

Missing 

No  Stat 

Total 

Page 

Dead 

Code  Elimination 

89 

11 

3  3 

0 

0 

17 

C-38 
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Summary 

over  all 

groups  :  product  data 

-  product 

model 

— 

Pairwise  Comparisons 

:  tot-.l  n  - 

17 

Systems  | 

n:  1 

Sys  Factor:  | 

V  91 
17 
0.37 

t  91 

16 

1.60 

s  91 

17 

0.30 

ffl  91 

17 

0 . 30 

d  91 1 
"161 
2.371 

Mean 

Vari¬ 

ation 

V  91 

n:  1 

16 

17 

17 

16  1 

Sys 

Factor:  | 

0.37 

0.37 

0.37 

0.37  1 

0.0% 

t  91 

n:  1 

16 

16 

16 

16| 

Sys 

Factor:  j 

1.60 

1.60 

1.60 

1.601 

0.0% 

s  91 

n:  1 

17 

16 

17 

16( 

Sys 

Factor:  | 

0.30 

0.30 

0.30 

0.301 

0.2% 

m  91 

n:  1 

17 

16 

17 

16  1 

Sys 

Factor:  ( 

0.30 

0.30 

0.30 

0.301 

0.0% 

d  91 

n:  1 

16 

16 

16 

16 

1 

Sys 

Factor:  | 

2.37 

2.37 

2.37 

2.37 

1 

0.0% 

-  Number  of  Test  Problems  in  the  Analysis 


No.  Problems  | 

v_91 

t_91 

s_91 

m_91 

d_91  1 

Possible 

application  | 

35 

30 

33 

29 

27  1 

37 

arithmetic  j 

22 

22 

22 

21 

21  1 

22 

classical  | 

38 

35 

38 

36 

33  I 

38 

data  storage  j 

13 

11 

6 

10 

10  1 

14 

data  structures  1 

50 

50 

50 

49 

46  1 

50 

delays  and  timing  j 

10 

10 

9 

11 

10  1 

12 

exception  Handling  j 

17 

17 

17 

16 

17  ' 

•  • 

generics  | 

5 

0 

5 

4 

0  I 

5 

input  output  t 

21 

18 

22 

18 

4  1 

23 

miscellaneous  j 

7 

6 

7 

6 

6  1 

7 

optimizations  | 

64 

65 

65 

63 

64  1 

65 

program  organization  j 

20 

20 

16 

18 

17  1 

20 

statements  j 

17 

17 

17 

17 

16  I 

17 

storage  reclamation  j 

51 

48 

41 

7 

41  1 

51 

subprograms  j 

16 

16 

16 

16 

16  1 

16 

systematic  compile  speed! 

89 

88 

80 

79 

24  1 

92 

tasking  j 

90 

89 

89 

87 

86  1 

102 

Total  1 

565 

542 

533 

487 

438  1 

588 
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Summary  over  all  groups  :  product  data  -  product  model 


Raw  Data:  | 

v_91 

t_91 

jj^ggni 

d_91 

1  Wgts 

application  | 

0.35 

1.32 

0.29 

0.33 

2.60 

1  1.0 

arithmetic  1 

0.43 

1.91 

0.24 

0.31 

2 . 08 

!  1.0 

classical  | 

0.39 

1.63 

0.30 

0.29 

2.47 

i  1.0 

data  storage  | 

0.35 

1.70 

0.37 

0.22 

1.79 

1  1.0 

data  structure! 

0.35 

1.71 

0.29 

0.29 

2.35 

1  1.0 

delays  and  timj 

0.30 

1.50 

0.30 

0 .31 

2.69 

1  1.0 

exception  hand! 

0.35 

1.48 

0.27 

0.29 

2 .53 

i  1.0 

generics  | 

1.26 

0.81 

0.84 

1  1.0 

input  output  1 

0.50 

2.71 

0.32 

0.37 

2.26 

1  1.0 

miscellaneous  j 

0.28 

1.38 

0.27 

0.29 

2.74 

1  1.0 

optimizations  | 

0.37 

1.69 

0.33 

0.29 

2.30 

1  1.0 

program  organi 1 

0.31 

1.40 

0.29 

0 . 29 

2.53 

1  1.0 

statements  I 

0.41 

1.90 

0.28 

0.29 

2 .13 

1  1.0 

storage  reclamj 

0.31 

1.14 

0.29 

0.22 

2.33 

1  1.0 

subprograms  i 

0.40 

1.75 

0.32 

0.32 

2.15 

1  1,0 

systematic  corn! 

0.62 

1.55 

0.62 

0.45 

2.23 

1  1.0 

tasking  j 

0.27 

1.35 

0.30 

0.32 

2.73 

1  1.0 

-  Outlier  Statistics: 

residual 

*  system 

factor 

*  row 

mean  >  actual 

Bounds 

Expect 

Got  1 

v_91 

3R|||| 

HQ| 

m_91  d_91 

—  Very 

2 

1  1 

1 

■ 

0 

0 

Low 

2 

3  1 

2 

0 

0 

0 

1 

+  High 

2 

0  1 

0 

0 

0 

0 

++  Very 

High:  1.33 

2 

8  1 

2 

1 

3 

2 

0 

Totals 

: 

8 

12  1 

5 

1 

3 

2 

1 

Residuals | 

v_91 

t_91 

s_91 

m_91 

d_91 

1  Means 

applicati | 

0.98 

0.84 

0.99 

1.14 

1.12 

1  0 

.98 

arithraeti j 

1.19 

1.20 

0.82 

1.03 

0.89 

1  0 

.99 

classical j 

1.05 

1.00 

1.00 

0.95 

1.03 

1  1 

.02 

data  stor| 

1.08 

1.20 

1 .40++ 

0.83 

0.86 

1  0 

.88 

data  struj 

0.97 

1.07 

0.98 

0.97 

0.99 

1  1 

.00 

delays  an j 

0.80- 

0.92 

0.98 

1.02 

1.12 

1  1 

.02 

exception | 

0.96 

0.94 

0.94 

0.98 

1.09 

1  0 

.98 

generics  j 

3.55++ 

2.81++ 

2 .91++ 

0.77- 

1  0 

.97 

input  out! 

1.12 

1.38++ 

0.87 

1.00 

j  1 

.  Z3 

misceXlan | 

0.77- 

0.87 

0.93 

0.98 

1.17 

1  0 

.99 

optimizat  j 

1.00 

1.06 

1.11 

1.00 

0.98 

1  0 

.99 

program  oj 

0.89 

0.91 

1.02 

1.02 

1.10 

1  0 

.97 

statement  j 

1.11 

1.18 

0.94 

0.99 

0.90 

I  1 

.00 

storage  rj 

0.98 

0.83 

1.13 

0.88 

1.15 

1  0 

.86 

subprogra  j 

1.12 

1.10 

1.09 

1.10 

0.92 

1  0 

.99 

systemati | 

1.54++ 

0.89 

1.90++ 

1.37++ 

0.86 

1  1 

.09 

tasking  j 

0.74  — 

0.85 

1.02 

1.09 

1.16 

1  1 

.00 

Sys  Fact  1 

0.37 

1.60 

0.30 

0.30 

2.37 

I 

Compile 

and  Link 

-  20  Dec 

1991  12:14: 

35  -  Page 
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Summary  over  all  groups  :  product  data  -  product  model 


-  System  Names  and  Descriptions 


v_91 

—  Host: 
t_91 

—  Host: 
s_91 

—  Host: 
m_91 

—  Host: 
d_91 

—  Host: 


VAXstation  3100 
VAXstation  3100 
MIPS  R2000A/R3000 
DECstation  3100  MIPS  RISC 
VAX  6220 


Target : 
Target : 
Target : 
Target : 
Target : 


VAXstation  3100 
VAXstation  3100 
MIPS  R2000A/R3000 
DECstation  3100  MIPS  RISC 
1750A 


-  System 

Factors  and 

Confidence 

Intervals  (including  graph) 

Systems 

Low 

Mean 

High 

Ratio 

1  0.3 

2.6 

-  1 

V  91 

0.32 

0.37 

0.42 

1.00 

1  l  +  l 

t  91 

1.42 

1.60 

1.80 

4.37 

!  1— i 

s  91 

0.27 

0.30 

0.32 

0.81 

l  +  l 

m  91 

0.28 

0.30 

0.31 

0.81 

l  +  l 

d  91 

2.14 

2.37 

2.62 

6.47 

1 

1  — + - j 

-  Significant  Diff  •  *  |  -  Data  Summary:  Total  n  -  588 


v_9  t_9  s_9  m  9  d  9  1  Gps  Valid  NoData  Comp  RunTim  Exclu  Other 


V  91 

* 

it 

★ 

* 

17 

565 

9 

0 

0 

14 

t“91 

* 

it 

★ 

* 

16 

542 

43 

0 

0 

3 

s_91 

★ 

* 

17 

533 

55 

0 

n 

0 

m  91 

* 

★ 

* 

17 

487 

101 

0 

n 

V 

d  91 

* 

it 

* 

* 

16 

438 

148 

0 

2 

-  Group  Weights 


application  1.0 
data_storage  1.0 
exception_handlin  1.0 
miscellaneous  1.0 
statements  1.0 
systematic_compil  1.0 


arithmetic  1.0 
data_structures  1.0 
generics  1.0 
optimizations  1.0 
storage_reclamati  1.0 
tasking  1.0 


classical  1.0 
delays_and_timing  1.0 
input_output  1.0 
program_organizat  1.0 
subprograms  1.0 
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o  o  o 


Summary 

over  all 

groups  :  product  data 

-  product 

model 

— 

Pairwise  Comparisons 

:  total  n  - 

17 

m 

Systems  | 

n:  1 

Sys  Factor:  [ 

V  91 
17 

1.08 

t  91 

17 

1.24 

s  91 
~15 
0.31 

m  91 

17 

0.58 

d  91| 
17  1 
1.591 

V  91 

n:  1 

17 

15 

17 

17  1 

Sys 

Factor:  j 

1.08 

1 . 09 

1.08 

1.081 

0.2% 

t  91 

n :  1 

17 

15 

17 

17  1 

Sys 

Factor:  j 

1.24 

1.24 

1.24 

1.24  1 

0.0% 

s  91 

n:  1 

15 

15 

15 

15  1 

Sys 

Factor:  j 

0.31 

0.31 

0.31 

0.31  1 

0.0% 

m  91 

n:  1 

17 

17 

15 

17  1 

Sys 

Factor:  | 

0.58 

0.58 

0.61 

0.581 

1.1% 

d  91 

n:  1 

17 

17 

15 

17 

1 

Sys 

Factor:  | 

1.59 

1.59 

1.65 

1.59 

1 

1.0% 

-  Number  of  Test  Problems  in  the  Analysis 


No.  Problems  | 

v_91 

t_91 

s_91 

m_91 

d_91  1  Possible 

application  | 

66 

76 

52 

61 

40  1 

86 

arithmetic  j 

108 

111 

39 

107 

108  1 

112 

classical  | 

83 

80 

82 

79 

54  j 

83 

data_storage  | 

91 

79 

27 

75 

70  1 

91 

data  structures  | 

224 

208 

135 

215 

204  1 

225 

delays^and  timing  | 

26 

16 

16 

25 

26  1 

41 

exceptTon_Randling  | 

58 

52 

39 

48 

39  1 

58 

generics  | 

19 

19 

7 

19 

15  1 

24 

input  output  1 

104 

94 

55 

93 

19  1 

108 

miscellaneous  j 

16 

16 

0 

16 

16  1 

17 

optimizations  | 

304 

304 

168 

288 

299  1 

305 

program  organization  | 

74 

74 

54 

72 

5  1 

74 

statements  j 

80 

80 

57 

80 

77  1 

80 

storage  reclamation  | 

47 

53 

46 

50 

29  1 

60 

subprograms  | 

79 

74 

36 

79 

79  1 

79 

systematic  compile  speed! 

73 

71 

0 

59 

20  1 

75 

tasking  | 

87 

89 

81 

78 

75  1 

109 

Total  1 

1541 

1496 

894 

1444 

1175  1 

1627 

Execution  Times  - 

20  Dec 

1991  12: 

14:16 

Page  3 
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Summary 

over  all 

groups  :  product  data  - 

p ,  -  iuct 

model 

Raw  Data:  I 

v_91 

t_91 

s_91 

m_91 

d_91 

1  Wgts 

application  { 

1.11 

1.35 

0 ,31 

0.44 

1.64 

i  1.0 

arithmetic  | 

0.76 

1.06 

0.25 

0.59 

1.42 

1  1.0 

classical  j 

0.85 

1.11 

0.40 

0.79 

1.91 

1  1.0 

data  storage  1 

0.75 

0.84 

0.27 

1.00 

1.04 

i  1.0 

data  structure! 

0.88 

0.95 

0.25 

0.65 

1.57 

i  1.0 

delays  and  timj 

0.64 

1.11 

0.50 

0.54 

1.29 

1  1.0 

exceptTon  Hand j 

1.26 

0.82 

0.33 

0.37 

1.18 

1  1.0 

generics  j 

0.90 

0.99 

0.11 

0.24 

1.86 

i  1.0 

input  output  i 

1.05 

1.47 

0.20 

0.35 

0.12 

1  1.0 

miscellaneous  j 

1.05 

1.30 

0.43 

1.02 

1  1.0 

optimizations  | 

0.93 

1.44 

0.21 

0.51 

1.23 

1  1.0 

program  orqani j 

1.11 

1.20 

0.28 

0.86 

1.78 

1  1.0 

statements  | 

1.04 

1.35 

0.19 

0.49 

1.56 

1  1.0 

storage  reclamj 

1.05 

1.20 

0.43 

0.71 

1.49 

j  1.0 

subprograms  j 

0.99 

0.98 

0.16 

0.46 

1.63 

1  1.0 

systematic  comj 
tasking  j 

1.21 

1.11 

0.46 

1.28 

1  1.0 

1.54 

0,94 

0.51 

0.46 

1.28 

1  1.0 

-  Outlier 

Statistics : 

residual  *  system  factor  *  row 

mean  «  actual 

Bounds  Expect 

Got  1 

< 

! 

yo 

t_91 

s_91 

m_91 

d_91 

—  Very 

Low  ; 

4  1 

0 

0 

2 

1 

1 

Low 

0.67  2 

1  1 

0 

0 

1 

0 

0 

+  High 

1.53  2 

0  1 

0 

0 

0 

0 

0 

++  Very 

High: 

4  1 

0 

1 

2 

1 

0 

Totals 

; 

8 

9  1 

0 

1 

5 

2 

1 

Residuals | 

v_91 

r-* 

1 

s_91 

m_91 

d_91 

1  Means 

applicati | 

1.06 

1.12 

1.02 

0.77 

1.07 

1  0.97 

arithmeti j 

0.86 

1.05 

0.98 

1.25 

1.10 

1  0.82 

classical  1 

0,78 

0.88 

1.25 

1.35 

1.19 

1  1.01 

data  storj 

0.89 

0.87 

1.11 

2.21++ 

0.84 

1  0.78 

data  struj 

0.95 

0.89 

0.93 

1.30 

1.15 

1  0.86 

delays  anj 

0.73 

1.10 

1.94++ 

1.14 

0.99 

i  0.81 

exceptTon ( 

1.46 

0.84 

1.34 

0.81 

0.94 

1  0.79 

generics  ( 

1.01 

0.98 

0.41  — 

0.51  — 

1.43 

1  0.82 

input  outj 

1.52 

1.86++ 

0.99 

0.95 

0.12— 

1  0.64 

misceXlan  j 

1.02 

1.11 

0.79 

0.68 

1  0.95 

optimizat  j 

0.99 

1.35 

0.78 

1.01 

0.90 

1  0.86 

program  o| 

0.98 

0.93 

0.86 

1.41 

1.07 

1  1.05 

statement  j 

1.04 

1.18 

0.66- 

0.91 

1.06 

1  0.93 

storage  rj 

0.99 

0.99 

1.41 

1.25 

0.96 

1  0.97 

subprogra  j 

1.08 

0.94 

0.59— 

0.94 

1.21 

1  0.84 

systemati j 

1.10 

0.88 

0.78 

0.79 

1  1.02 

tasking  j 

1.50 

0.81 

1.72++ 

0.83 

0.85 

1  0.95 

Sys  Fact  1 

CD 

O 

• 

1.24 

0.31 

0.58 

1.59 

1 

Execution 

Times 
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1991  12:14:16 
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Summary  over  all  groups  :  product  data  -  product  model 


-  System  Names  and  Descriptions 


v_91 

—  Host: 
t_91 

—  Host: 
s_91 

—  Host: 
m_91 

—  Host: 
d_91 

—  Host: 


VAXstation  3100 
VAXstation  3100 
MIPS  R2000A/R3000 
DECstation  3100  MIPS  RISC 
VAX  6220 


Target 

Target 

Target 

Target 

Target 


VAXstation  3100 
VAXstation  3100 
MIPS  R2000A/R3000 
DECstation  3100  MIPS  RISC 
1750A 


-  System 

Factors  and 

Confidence 

Intervals  (including  graph) 

Systems 

Low 

Mean 

High 

Ratio 

I  0.2 

1.8 

V  91 

0.94 

1.08 

1.25 

1.00 

1— +— i 

- -  ..  I 

t  91 

1.09 

1.24 

1.40 

1.14 

1  1— + - 1 

s  91 

0.23 

0 . 31 

0.43 

0.29 

i  “••• —  1 

ra  91 

0.46 

0.58 

0.74 

0.54 

d  91 

1.39 

1.59 

1.82 

1.47 

1  1- 

- + - 1 

Significant 

Diff 

a  * 

— 

Data 

Summary: 

Total 

Bmmmm 

n  a* 

1627 

v_9  t_9 

s_9 

m_9  d_9 

GpS 

Valid 

NoOata 

Comp  RunTim 

Exclu 

Other 

V  91 

* 

*  * 

17 

1541 

0 

14 

1 

25 

46 

t  91 

- 

* 

*  — 

17 

1496 

29 

31 

12 

1 

58 

s  91 

it  * 

*  * 

15 

894 

30 

82 

15 

2 

604 

m  91 

*  A 

* 

★ 

17 

1444 

81 

42 

21 

0 

39 

d  91 

★  .. 

* 

A 

17 

1175 

319 

71 

22 

1 

39 

-  Group  Weights 


application  1.0 
data_storage  1.0 
exception_handlin  1.0 
miscellaneous  1.0 
statements  1.0 
systematic_compil  1.0 


arithmetic  1.0 
data_structures  1.0 
generics  1.0 
optimizations  1.0 
storage_reclamati  1.0 
tasking  1.0 


classical  1  3 
delays_and_timing  1.0 
input_output  1.0 
program_organizat  1.0 
subprograms  1.0 
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AES  Review  Outline 


Background 
Taat  Hamosa 
Spacifle  Tast  Problams 
Asaaaaora 
Analyala/  Raporting 


AES  2.0  REVIEW  BACKGROUND 


Raad  documentation  and  coda  for  performance  tests 
Executed  the  performance  tast  groups 
Ravlawad  documentation  and  ran  other  selected  groups 
AES  design  philosophy  assumes  testing  service 
Expected  to  develop  core  of  people  experienced  with  porting 
Relatively  small  number  of  performance  tests 
No  automated  system  comparison  tool 
Emphasizes  tsxtual  reporting 
Oualltatlva  findings  (optimization  performed  or  not) 

AES  provides  broad  coverage  of  capabllHIes  of  ’Vrhole  APSE” 


Workshop 
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TEST  HARNESS  PORTING  EFFORT 


AES  forces  users  to  lesm  operating  system  interfaces  for 
Control  of  split  screen 
Spawning  procesaea 

Invoking  Job  control  statemems  from  Ada  program 
Requires  Information  about  AES  Internal  structures 
Requires  adaptation  of  preprocessor 
Desires  compiler  supporting  all  Ada  wKh  OS  Interfaces 


ACEC/AES  MERGER  WORKSHOP 
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r 

AES  EXAMPLE  1 :  TI01 D 

Test  problem  contains  40  statsmants  of  the  form 

Intended  to  test  efficiency  of  record  comparlslon 

SAME  :=  A1  =  A2  ;  -  where  A*  Is  a  record 

SAME  :=  A2  s  A3  ; 

NOT_SAME  :=  (A1  /=  A2)  and  (A3  /=  A4): 

NOT_SAME  :=  (AS  /=  A6)  and  (A7  /=  A8); 

L 

) 

FLAWS  IN  EXAMPLE  1 


38  of  40  asslgnnwnts  ar«  dead 
ALL  expressions  are  loop  Invariant 

Even  If  record  comparisons  could  raise  exceptions,  LRM  would 
explicitly  permit  reordering  (11.6) 


WarkMep 
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AES  EXAMPLE  2:  TO02 


Examptos  RL11  &  RL12  ara  intandad  to  dataet  whathar  common 
aubaxpraaaiona  involving  two^iimanaional  array  addrassing  for 
Intagara  ara  raeognbad  aa  common 

RL11  =>  RAA  (K.  L)  :=  RBB  (K.  L)  *  RV; 

RAA2  (K,  L)  :=  RBB2  (K.  L)  +  RV; 

RL12  =>  RAA  (K,  L)  :=  RAA  (K.  L)  4-  RV; 

RAA2  (K.  L)  :=  RAA2  (K,  L)  RV; 

AES  aasumaa  RL12  will  ba  amaiiar  than  RL1 1  iff  common 
subaxpraaaions  ara  racognizad 


Work  snap  BOGNO 


FLAWS  IN  EXAMPLE  2 


Array  TYPES  ara  Idantteal  so  tha  ALL  subscripting  axprassions  ara 
common 

All  subscripting  axprassions  ara  loop  invariant 
Usa  of  ADD_TO_MEMORY  instruction  can  confound  Intorpratatlon 
No  "cradit"  for  racognizing  that  axprassion  could  bo  ovaluatod  onco 
Conclusions  drawn  from  tast  rasults  can  ba  wrong 


wwkaiwp 


BOGNO 
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FLAWS  IN  EXAMPLE  3 


TRUE1  &  TRUE2  art  loop  Invariant 

Instruction  prafatchlng  will  favor  second  example 

The  Idea  that  a  aystam  has  a  constant  FOR  LOOP  overhead  Is 
fiswed 

An  optimizer  may  unroll  some  loops,  reducing  loop  overhead 

Complexity  of  body  may  permit/prevent  keeping  FOR  Index  In 
register 

Size  determIrMS  whether  code  can  use  long/short  format 
Instruct  Iona 

Memory  effects:  cache  and/or  prafatchlng  and/or  "loop  mode” 
Target  processor  may  have  idiomatic  instructions 
After  loop  Invariant  motion,  body  might  be  null 
Strength  reduction  on  FOR  loop  index  used  only  as  subscript 
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AES  EXAMPLE  4:  TR23 


Test  problem  to  detect  whether  compiler  does  loop  motion: 

for  1  In  ONE_TO_TEN  loop 
SUH(I):=S*  A(l): 

A  (INDEX)  ;s  S;  -  INDEX  le  out-of-range 
end  loop: 

Tests  whether  value  of  SUM(1)  has  been  modified 


inM3  13 


workshop  BOBNO 


FLAWS  IN  EXAMPLE  4 


"A(INDEX)  :=  S;"  only  Invariant  If  flow  analysis  deternitnes  INDEX 
Is  never  one 

Confounds  loop  Invariant  motion  with  data  flow  analysis 


workshop 
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ASSESSORS 

Includas  asMsaor*  for  rton-Ada  apacifle  capabilitlaa 

Raqulramanta  analyzar 

Varalon  Configuration  Control 

Editora  (ganaral  purpoaa) 

Command  Languaga  Intarpratar 

Includaa  assasaora  for  varioua  Ada*apacifie  capabllKlaa 

Compliar/LInkar  diagnoatica 

Compllar/LInkar  capacity  llmka 

Compllar  parformanca 

Runtime  parformanca 

Nama  axpandar 

Pratty  printer 

Sourca  ganarator 

Timing  analyaia  toola 

Taat  eovaraga  tool 

Stub  ganarator 

Aaaartlon  chackar 

Croaa  rafaranca  analyzar 

Syntax  orlantad  aditor 

Tastbad  ganarator 

J 
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ANALYSIS  /  REPORTING 


AES  do«s  not  provida  comparativa  anaiyaia  tool 
AES  raporta  stataa  eonelualona  without  supporting  data 
TO02  raporta  whathar  optimization  parformad  or  not 
AES  praaants  a  lot  of  daacrtpthra  taxt  along  with  raaulta 


V _ 

Work  Shop  BOeNO 


SUMMARY 


Test  Harness 

Inappropriate  to  nomtast  service  based  usage 
Test  Problems 

Many  contain  flaws  which  should  bo  corrected 
Assessors 

AES  provides  broad  coverage  of  APSE  capabllltites 
Analysis 

Lack  of  automated  system  comparisons  Is  significant 


V _ 
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Settwan  EngInMriog  IratlUil* 


Comments  on  the 
Ada  Evaluation  System 


January  22, 1992 

Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh  PA  15213 

Sponsored  by  the  U.S.  Department  of  Defense 


Ctnwei  tint  Uwiin><^r 

Software  Cnglnaarlng  Inatliuta _ 

Background 

The  SEi  has  had  access  to  the  AES  for  about  four  years, 
and  has  used  it  in  several  ways: 

•  As  software  subjected  to  critical  review 

•  As  an  element  in  a  benchmark  tutorial 

•  As  an  evaluation  tool  in  performance  analysis 

The  AES  elements  considered  in  this  talk: 

•  Executable  benchmark  tests  (primary) 

•  Test  harness  (primary) 

•  Check  lists  (secondary) 

•  Documentation  (secondary) 
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Seltwan  EnglmWing  InaUlut* 


Organizational  Issues 


Consistent  use  of  Test  Categories 

•  Tests  are  organized  into  groups 

1 .  Object  under  test  (compiler,  loader,  etc.) 

2.  Concept  under  tests  (optimizations,  etc.) 

•  Categories  are  used  for 

-  Documentation 

-  File  Naming 

-  Testing 

-  Reports 

Convenient  Access  to  Data  (Database) 

•  Data  stored  as  text 

•  Database  used  for  results  and  control 
variables 

•  Only  one  value  per  database  item  (e.g.  one 
compiler  per  database,  one  test  run  only) 


Settwari  EntlMlirlng  Iratitutt 


File  Organization 


4 


Targtl  Syctam 


AES  (Upon 
Oanaraler 
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SoWwifi  Englii— ring  ImWmO 


User  Interface  -1 

Strengths 

•  Interactive 

•  Integrates  all  important  components 

-  Set  up 

-  Database 

-  Testing 

-  Report  generation 

•  Automatic  or  manual  mode  using  same  model 

•  Script  driven,  flexible 


C«nn»»  t>li«  Uni  I  iiw^i 

SoWww  EnglfWfIng  IntUttrti _ 

User  Interface  -2 

Weaknesses 

•  Modal  (set  up  mode,  test  mode,  report  mode) 

•  Undocumented  commands 

•  Partial  memory  of  previous  state 

•  Portability  complicated  by  screen  interface 

•  Weak  heip  and  tutorial 

•  Database  access  Is  limited  to  raw  data 

•  No  provision  for  subsetting  or  exchanging 
data 
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SoWwK  Englwortna  ImiWiw 


Scripts 

Scripts  provide 

•  Conditional  testing  based  on  test  results 

•  For  user  modifications 

•  Convenient  substitution  points  for  system 
dependencies 

Script  problems 

•  Hard  to  identify  truly  universal  operations  (e.g. 
Verdix  does  listing  with  a  separate  utility,  not 
through  compiler) 

•  Sometimes  no  handle  provided  (e.g.  Verdix 
uses  “.a*'  for  all  source  files) 

•  Operations  may  be  divided  or  combined  (e.g. 
optimization:  pragma,  command  iine  switch  or 
both) 


Focused  Testing 


Some  of  my  favorite  tests/methods: 

•  Limit  testing  (using  binary  chop) 

•  Memory  allocation  strategies  (three  models) 

•  Variation  in  implementation  (e.g.  case 
impiementatlon  strategies) 
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SoHwan  ffigbiMring  iMtttirtt 


Test  Issues 

Identified  test  needs 

•  Measuring  individual  features 

•  Compare  different  systems 

•  Compare  different  versions 

•  Time  statements 

•  Coding  for  performance 

AES  vs.  ACEC 

•  Measuring  individual  features;  AES  better 

•  Compare  different  systems:  AES  worse 

•  Compare  different  versions:  AES  equal 

•  Time  statements:  AES  much  worse 

•  Coding  for  performance:  AES  better 


Cvntelv  IiMm  IMwMy 

Sefiwart  Cnginatring  iMtKuto _ 

Timing 

Features 

•  Ignores  certain  biases 

•  Selects  timing  based  on  clock  resolution 

•  Portable,  only  needs  standard  clock 

•  Tests  for  consistency  of  results 

•  Time  value  is  average  of  ail  tests,  not 
minimum 

Issues 

•  No  provision  for  automated  substitution 

•  Not  suited  for  fast  timers 

•  Can  only  generate  average  values 

•  Timing  can  start  before  I/O  concludes 


C-58 


ACEC/AES  MERGER  WORKSHOP 


SoHwiw  tinglnwing  IniUtuf 


Bias  Control 

AES  controls  for  certain  kinds  of  bias: 

•  Memory  location  effects:  averaged  values  by 
multi-statement  segments 

•  TinrJng  overhead:  tested  code  segments  kept 
large,  timing  overhead  is  assumed  to  be  small 

•  Timing  variation:  multiple  runs,  tests  which 
show  significant  variation  are  marked  as 
failing 


C«nv*«  IMm  UtiMiMii 
SettwaraC^lMtrtng  Inttliirt* 


Missing  but  Desirable 


•  Open  systems:  features  and  documentation  to 
support  customization  and  extension 

•  Presentation:  flexible  report  and  graphic  output 
•As  part  of  standard  presentations 

-To  allow  Interactive  testing 

•  Data  manipulation:  need  to  select  and  exchange  data 
-For  spreadsheets 

-Raw  data  for  statistical  packages 
-In  table  format  for  text  processing 
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Cfi0lnMrtfi9  ImiNmip 


Comments  on  the 
Preliminary  Release 
of  ACEC  3.0 


22  January  1991 

Software  Engineering  Institute 
Carnegie  Meiion  University 
Pittsburgh  PA  15213 

Sponsored  by  the  U.S.  Department  of  Defense 


Sottwire  Enjln»«rln9  IntUlut* 

Background 

SEI  has  been  invoived  with  ACEC  for  the  past  few  years 

Reiease  2.0  has  been  used  in  the  past  year 

•  As  software  subjected  to  critical  review 

•  As  an  element  in  a  benchmark  tutorial 

•  As  an  evaluation  tool  in  performance  analysis 

Preliminary  release  3.0  was  received  in  mid-December, 
1991,  so  an  extensive  evaluation  of  it  has  not  been 
performed 
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C«MV*  IMm  Upwanay 
SoNwm  EnglnMrtng  InaiHuto 


SEI  Work  to  Date  with  Release  3.0 


All  documents  read  and  all  pre-test  steps  run 
Limited  number  of  performance  tests  run 
Problems  encountered  running  SSA  and  Menu 
CA  and  assessor  tools  not  run 
SEI  Configuration 

•  Compiler:  Verdix  VADS  VMS  -  MC68030  6.0.5(f) 

•  Host:  DEC  MIcroVAX  3200  running  VMS  5.3 

•  Target:  25  MHz  Motorola  MC68030 

•  Compiler  for  host-specific  analysis  tools  was  DEC 
VAX  Ada  2.1 


Softwai*  Cnglnaarlng  Insllliit* 

Overall  Comments 

The  User’s  Guide  and  the  organization  of  the  software 
are  much  improved  over  the  previ  's  release' 

•  Logical  step-by-step  guide  to  pri  tests 

•  Pre-tests  incorporate  actual  execution  of  tests  and 
analysis  tools 

•  Command  file  naming  conventions  and  division  of 
tests  into  performance  groups  makes  life  easier 

The  Reader’s  Guide  still  needs  work  to  achieve  the 
clarity  and  usefulness  of  the  User’s  Guide,  particularly 
the  sections  dealing  with  the  Comparative  Analysis 
tool 

The  pre-test  and  test  suite  command  files  do  not 
adequately  address  the  needs  of  host-target  systems 
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Comments  on  the  Reader’s  Guide 

The  Reader's  Guide  doesn’t  yet  clearly  answer  the 
question:  How  does  the  ACEC  compare  compiler  Cl 
on  machine  Ml  with  compiler  C2  on  machine  M27 

There  should  be  a  clear  statement  of  the  level  of 
knowledge  required  of  a  user  for  correct  interpretation 
of  all  analysis  tool  outputs- 

The  sections  on  the  Comparative  Analysis  tool's  output 
and  background  need  to  be  re-organized  and  made 
more  understandable 

The  chapter  on  timing  techniques  is  good  but  needs 
some  re-organization  to  make  it  more  user-friendly 


SoNwar*  EnglMwUo  Iratttut* 

Some  Problems  Encountered 

TCAL1  and  TCAL2  tests  (pre-test  step  4)  didn’t  work 

Double-precision  math  library  test  of  Power  function 
failed  with  an  Argument_Error  exception 

SSA  tool  couldn't  open  database  file  created  by  the 
Condense  tool;  Menu  program  subsequently  crashed 

Menu  program  crashed  Immediately  when  "PS:"  was 
specified  in  a  VMS  pathname  in  the  System  Names  file 
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Concluding  Remarks 

Provide  users  with  some  estimate  of  how  iong  it  takes 
to  set  up  and  run  the  ACEC  and  anaiyze  the  resuits 

Emphasize  the  need  for  users  to  treat  running  ACEC  as 
part  of  a  iarger  overail  evaiuation  PLAN 

Think  about  graphical  output  and/or  output  suitable  for 
analysis  by  spreadsheets 

Think  in  terms  of  benchmark  generation  rather  than 
benchmark  instantiation 
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Hypertext  Version  of  the 
E&V  Reference  System: 
A  Sampler 

23  January  1992 

[Sample  material  from  the 
User's  Guide  and  system  screens] 


Prepared  by: 

Bard  S.  Crawford 
TASC 

55  Walkers  Brook  Drive 
Reading,  MA 
01867 

617-942-2000 

crawford@ajpo.sei.cmu.edu 
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U««f^  Quid*  to  tho  Hyportnrt  Voraion  of  iho  E&V  Roforoneo  Systom 


1.  INTRODUCTION 

The  Ada  Programming  Support  Eavirotunent  (APSE)  Evaluation  and  Validation  (E&V) 
Reference  System  is  a  pair  of  documents  developed,  and  periodically  updated,  by  the  APSE  E&V 
Project,  sponsored  by  the  Ada  Joint  Program  Office  and  led  by  the  US  Air  Force  Avionics 
Directorate  of  the  Wright  Laboratory.  The  documents  are  entitled  the  "E&V  Reference  Manual" 
and  the  "^V  Guidebmk." 

The  E&V  Reference  Manual  provides  a  firamework  for  understanding  APSEs  and  their 
assessment,  and  establishes  common  terminolQgy.  One  chapter  discusses  an  APSE  as  a  whole  and 
its  assessment.  Other  chapters  are  indexes  to  APSE  component  characterization  and  assessment, 
organized  by  life  cjrcle  activities,  APSE  tool  category,  APSE  function,  and  attribute  to  be  assessed 
An  entry  in  an  index  consists  of  a  description,  cross  references  to  other  entries  in  the  Reference 
Manual,  and  cross  references  to  the  "E&V  Guidebook,”  The  manual  is  intended  to  help  a  variety 
of  users  obtain  answers  to  their  questions.  As  a  stand-alone  document  it  is  intended  m  help  a  u»r 
find  useful  information  about  index  elements  and  relationships  among  them.  In  conjunction  with 
the  Guidebook,  it  is  intended  to  help  users  find  criteria  and  metrics  for  assessment  of  APSEs  and 
their  con^ionents. 

The  E&V  Guidebook  provides  descriptions  of  specific  instances  of  assessment 
technology.  These  include  evaluation  (assessment  of  performance  and  quaUty)  and  validation 
(assessment  of  conformance  to  a  standard)  techniques.  For  each  category  of  item  to  be  asKssed 
(e.g.  compilation  system,  test  system,  whole  APSE,  etc.),  there  are  brief  descriptions  of  applicable 
tools  and  aids  —  such  as  test  suites,  questionnaires,  checklists,  and  structured  experiments  —  and 
references  to  primary  documents  containing  detailed  descriptions.  The  Guidebook  also  contains 
synopses  of  dncuTnennt  of  general  historical  importance  to  the  entire  field  of  Ada  environments  and 
their  assessment. 

Hard  copy  versions  (1.1,  2.0,  3.0)  of  both  documents  have  been  published  beginning  in 
1987.  These  are  available  thiwgh  the  Demise  Technical  Information  Center  (DTIC).  The  Version 
3.0  DTIC  numbers  are: 

AD  A236  697  -  E&V  Reference  Manual 

AD  A233  494  -  E&V  Guidebook 

The  text  of  the  hypertext  version  (3.1)  is  based  on  the  most  recent  hard  copy  version  (3.0), 
published  in  Febru^  1991  —  with  a  few  minor  additions  and  corrections.  The  hypertext  version 
rans  on  Macintosh®  computers  as  a  set  of  HyperCard®  stacks.  It  requires  Hypcrcarf  Version  2.0 
or  later.  It  is  shipped  as  a  set  of  three  3.5  inch  double-density,  double-sided  disks,  plus  this 
document 

Disk  A  ffonfini  g  rtarici ..  two  Special  stacks  and  part  of  the  E&V  Reference  Manual. 

Disk  B  ooutains  3  stacks  -  Oiapten  6  and  7,  and  Ai^ndices  of  the  E&V  Reference 

ManuaL 

Disk  C  contains  18  stacks  --  the  entire  E&V  Guidebook. 

E&V*Maps  and  E&V*Help  are  the  two  special  stacks  mentioned  above.  The  other  27  stacks 
correspond  directly  to  textual  materl  il  previously  published  in  the  most  recent  hard  copy  version 
(3.0) "  except  for  the  many  "hyperlinks”  and  navigation  devices  incorporated  along  with  the  text 
in  the  hypertext  version  (3.1). 
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2.  INSTALLATION  AND  START-UP 

You  must  have  a  Macintosh  with  Hypercard  Version  2.0  or  later  already  installed.  The 
E&V  Reference  System  stacks  require  approximately  1.9  megabytes  of  memory  on  your  hard  disk. 

It  is  very  easy  to  install  the  system  for  anyone  familiar  with  the  Macintosh  desktop  system 
for  creating,  copying,  and  dragging  folders  and  files.  Perform  the  following  steps  to  insudl  the 
system: 

1 .  Create  a  new  folder  with  a  name  such  as  "E&V  RefSys" 

2.  Copy  the  contents  of  disk  A  (8  files)  into  your  new  folder. 

3.  Copy  the  contents  of  disk  B  (3  files)  into  your  new  folder. 

4.  Copy  the  contents  of  disk  C  (18  files)  into  your  new  folder. 

5.  AiTan^the29  icons  representing  the  29  files  in  a  neady 
orgaidzed  manner  such  as  diat  shown  in  Fig.  2-1. 

To  start  the  system  running,  you  can  double-click  on  any  of  the  29  icons.  Normally,  you 
will  want  to  start  from  a  high-level  view.  The  way  to  do  this  is  to  double-click  on  the  E& V-Maps 
icon.  This  takes  you  to  the  first  card  in  the  staclc  it  is  called  Top-Level  E&V  Map.  If  you  are 
already  familiar  with  the  system  based  on  past  experience,  you  may  want  to  go  direcdy  to  one  of 
the  chapters  by  double-cUcldng  on  the  icon  corresppnding  to  that  chapter.  You  can  easily  get  to  the 
help  screens  tom  every  card  in  any  of  the  other  28  stacks,  by  clicltog  on  the  ?  butuxi.  You  can 
also  start  in  the  E&V-Hdp  stack  by  double-clicking  on  the  icon  with  that  name.  In  fact,  a  good 
way  to  begin,  if  this  is  your  first  look  at  the  system,  is  to  go  direcdy  to  the  E&V-Hdp  stack  and 
browse  through  the  first  secdon  called  Welcome  and  Introduction.  The  help  screens  are  also 
printed  out  in  Section  3  of  this  User's  Guide. 


vm 

inuirTiiffi 

29  dMns 

^E&V-Maps 

^RM-Front 

^0B>Front 

^GB-Ch10 

^E&V-Htlp 

^RM-Ch1 

^GB-Ch1 1 

0RM-Ch2 

^6B-Ch2 

^GB-Ch12 

^RM-Ch3 

^GB-Ch3 

^GB-Ch13 

^RM-Ch4 

^GB-CM 

^GB-Ch14 

^RH-ChS 

0GB-Ch5 

^GB-Ch15 

0RM-Ch6 

0GB-Ch6 

^GB-Ch16 

^RN-Ch7 

^GB-Ch7 

^GB-End 

^RH-En4 

^GB-Ch8 

0GB-€h9 

Figure  2- 1  Suggested  Arrangement  of  Stack  Icons 
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3.  HELP  SCREENS:  PRINT-OUT 


This  section  provides  a  print-out  of  the  complete  set  of  help  screens  in  the  E&V-Help 
stack.  The  tirst  card  in  the  stack  is  the  help  Main  Menu.  The  other  cards  are  ^uped  into  six 
subdivisions,  as  indicated  in  the  Main  Menu.  The  Main  Menu  is  printed  out  seven  times  in  the  next 
four  pages,  each  time  with  a  different  choice  of  "pop-up  fiel^"  displayed.  This  provides,  in 
effect,  a  Table  of  Contents  for  the  complete  set  of  help  screens.  The  remainder  of  the  cards 
provide  an  overview  of  the  contents  of  the  entiie  system  aM  how  to  use  it 


E&V  H«1p  Vou  should  b*  in  Usor  Ltvol  1 :  Browsing  —  s»o  Iasi  card  of  Homo  stack 
Uso  arrow  buttons  at  bottom  to  go  forward  or  backward. 

Main  Menu 


[B  Welcome  end  Introduction 

(21  Navigating  with  “E&V  Maps" 

SI  Other  Navigation  Aids 

31  Using  the  Formal  Chapters 
(RM  4-7,  GB  4-16) 

S  Early  Chapters  and  Appendices 

(S  Marking,  Printing,  and  User 
Feedback 


Click  the  text  of  a  Mein  Menu  item 
to  get  e  list  of  subtopics. 


Click  a  small  box  to  go  di  recti  g  to 
section  of  help. 


Quit  E&V  Helpl 
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E&V  Help 


Veleeme  artd  Intreduetien 


la  Welcome  Text 

(Note:  be  sure  to  click  the  “bslloen"  and  then  click  the  "close  box“) 


Welcome  to  the  HyperCard  version  of  the  E&V  Reference  System  — 
Version  3.1. 

The  system  helps  users  make  assessments  of  tools,  tool  sets,  and 
environments  (APSEs)^^  It  contains  two  electronic 
"hyperdocuments:" 

~E&V  Reference  tianuar 
*E&V  Guidebook.* 

Assessments  fall  Into  two  categories: 

Evaluation  (E)  is  assessment  of  performance  and  quality. 
Validation  (V)  is  assessment  of  conformance  to  a  standard. 


Quit  E&V  Help 


e&v  Help 


Veleome  and  Introduction 


la  Welcome  Text  -  continued 


Hard  copy  versions  (1.1,  2.0,  and  3.0)  have  been  published  beginning  in 
1987.  These  are  available  through  the  Defense  Technical  Information 
Center  (DTIC).  The  Version  3.0  DTIC  Numbers  are: 

E&V  Reference  Manual  —  AD  A236  697 
E&V  Guidebook  —  AD  A236  494 

This  electronic  version  (3.1)  is  based  on  the  most  recent  hard  copy 
version  (3.0),  published  In  February  1991.  Most  of  the  text  is  the 
some  —  the  exceptions  to  this  rule  ore  indicated  in  "Version  3.1 
Upgrades"  (  see  E&V  Help  Item  Ih).  The  early  chapters  of  both 
documents  contain  background  material  on  the  need  for  E&V  and  the 
history  of  the  E&V  Project. 


Quit  E&V  Help 
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la  Welcome  Text  -  continued 


The  next  card  is  a  pictorial  representation  of  the  two  documents  and 
their  relationship  to  one  another. 

Other  graphical  representations  of  the  system  may  be  found  in  the 
stack  called  E&V  flaps  —  a  very  important  stack,  which  you  should 
be  sure  to  read  about  a  little  later  in  this  E&V  Help  stack. 

E&V  Help  and  E&V  Maps  are  of  course  not  included  in  the  hard  copy 
version  of  the  system. 


Quit  E&V  Helpl 


E&V  Help 


Vtleome  and  Introduction 


fb  Welcome  Diagram 

Welcome  to  the  HyperCard  vereion  of  the  "E&Y  Reference  Syetem," 

The  system  consists  of  tvo  documents,  vhich  contain  many  inter-document 
and  intra-document  links  (pointers),  as  indicated  pictorially  below. 

E&V  Reference  Menmi  (RM) 
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4.  E&V  MAPS:  PRINT-OUT 

This  section  provides  a  piint-out  of  the  eight  cards  of  a  special  stack  called  E&  V-Maps. 
The  on-screen  versions  of  most  of  these  cards  provide  a  great  deal  of  hidden  information  — 
available  in  pop-up  fields.  You  will  not,  of  course,  have  access  to  the  hidden  information  in  this 
printed  form.  But,  you  can  see  how  the  top-level  access  is  organized,  and  you  can  read  about  the 
mechanisms  involved  by  reviewing  Part  2  of  &e  E&V-Help  stack  given  previously. 


E&V  Maps 


Top-Level  E&V  Mop 


E&V  Reference  Manual  (RM) 


(Title  O'  Front  Matter) 
C  Early  Chapters  ') 


Schema  Map 


PftlntTS 


(  Appendices,  etc,  j 


Click  In  rectangular  buttons  to  go  to  maps. 
Click  in  rounded  buttons  to  see  more  details. 


E&V  Guidebook  (6B) 


(Title  e  Front  Matter) 

(  Eariu  Chapters  ) 
SynopsBS  (CJt4) 


C  Appendices,  etc. ) 


Click  in  numbers  or  italicized  words 
to  go  directig  to  chapter,  appendix,  etc. 


ACEC/AES  MERGER  WORKSHOP 


C-71 


UMr^OuUvtoth*  Hyp»nw»  Vfioff  of  th>  EAV  Rofwnno  Syittn 


E&V  Maps 


Functions  hap 

Guide  to  Chapter  7.  FtiBctlan  of  the  E&Y  Reference  Manual 


7.1  TrAMfarmatlan 

~7  1.1  Editing  ^ 

7.1.2  Formatting  ^ 

7.1.3  On-Line Aaalatanca 

7.1.4  Sort/Merga 

7.1.5  Graphics  Generation 

7.1.6  Translation 

7.1.7  Sgnthesis<^ 


Chek  numbor  to  qo  to  soctton.  ' — 

Click  function  name  to  so#  description. 

Click  round  button  to  soo  lovor-lovot  details, 


icTTnornKiiiJirTT 


7.2  Ma— genaeat _ 

7.2.1  information  Management 

7.2.2  Project  Management 

7.2.3  Computer  System  Mgmt  m 


7.3AMljsia _ 

7.3.1  Static  Analysis  wz — 

7.3.2  Dynamic  Analysis 

7.3.3  Formal  Yaiificatlon 

7.3.4  Problem  Report  Analysis 

7.3.5  Change  Reguest  Analysis 


E&V  Maps 


Assessors  hap 

Guide  to  Chapters  5- 1 6  of  the  E&Y  Guidebook 


5.  General  Purpose 
Assessors 


6.  Compilation  System 
Assessors 

7.  Target  Code  Generation  Aids 
and  Analysis  Toolset  Assessors 

8.  Test  System  Assessors 

9.  Tool  Support  Component  m 
Assessors 

1 0.  Regulremants/Design  ^ 
Support  Assessors 


1 1 .  Configuration  Management^ 
Support  Assessors 

1 2.  Distributed  System  Dev’mt  ^ 
and  RTS  Assessors 

1 3.  Distributed  APSE  Assessors 

14.  Whole  APSE  Assessoprs<l^ 

1 5.  Information  Management^ 
Support  Assessors 

16.  Other  Assessors^ 


CHek  number  to  go  to  section. 

Click  round  button  to  see  lever-level  details. 


^  L' j  'B  f -I  ij^M  ll^m  m  ^ 
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CtMpttr  7  Functions 


7. 1.6.7  Compilation 

Description: 


Compilation:  T ranalating  a  eomputar  propram  axpraaaatf  1  n  a  procedural  or 
problem-oriented  language  Into  object  coda.  I  oKaan  1 985] 


■ 


Cross  References:  I 

Guidebook  References: 


Life  Cycle  Activltlea 


Tools 
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6.7  UK  Ada  Evaluation  System  (AES) 


I  |i|i  I  1 1  iMi  iihii  iii|  I  u  \i  hi  I  I  iMii|  I  I  I  1 1 II  I  I  I  II 11 


UK  Add  EUflLUATION  SVSl^  <flES> 


IlilliiTilUl 


Purposa:  Evaluation  of  Ada  coapHors  and  assoc iatad  I  inkars/loodars,  program 
I  ibror^  si^taas,  dabuggars,  e»>d  run-tiaa  I  ibrorias.  A  last  suita  and  a 
mathodoiogg  <AES)  warm  davaiopad  bg  Softaora  Sciancas  Ltd.,  undar 
sponsorship  of  tha  UK  rtinistrg  of  Dafansa  <no0>.  Tha  British  Standards 
Instituta  <BSI )  has  baan  sponsorad  bg  tha  hoO  to  provida  on  Ado  Evaluation 
Sarvica,  using  tha  AES.  Intarastad  parties,  such  as  compiler  vendors  or 
potential  coopllar  purchasers,  nag  pag  BSI  to  conduct  an  evaluation  or  to 
suppig  a  copg  of  an  existing  asKiluation  report. 


(tRH:  Cenpl  lotion 


7. 1.6.7,  Accurocg 

cm:  Anonaig  Manoganant 
cm:  Copocitg 
cm:  Cost 
cm:  Oparobi  I  i  tg 


6.4.1, 

6.4.2, 
6.4.6, 
6.4.  11, 
6.4.21, 


Processing  Effectiveness  6.4.23, 


iiJLfil 


Level  E&U  Me 
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ACEC  -  AES  Merger 

Objectives 

interface 

Performance  Tests 

Analysis 

Assessors 

V 

J 

Work  Shop 

BOEING 

1/1M2  1 

Objectives 

Portability 
Ease  of  adaptation 
Ease  of  use 

Minimize  cost/benefit  ratio  for  users 
Upward  compatabiiity  (200  ACEC  users) 


Take  the  best  of  the  AES  and  add  it  to 
the  ACEC 


Resource  Constraints 
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Interface  Issues: 
The  Test  Harness 


Database 


Preprocessor 
Command  file  generation 
Ada  program  generation 


Direct  execution  mode 


WohcStiep 


BOGMQ 


Database 


Keeps  track  of  progress 


Insure  that  checkout  tests  have  been  run 


Using  information  from  checkout  tests 


Warfe«M» 


BOBNO 


i/iMa  3 


1/1«M  4 


C-78 


ACEC/AES  MERGER  WORKSHOP 


Command  File  Generation 


Ease  the  adaptation  process 

Do  not  have  to  remember  file  names 

Tailoring 


Database  information 
User  requests 


WofkHiep 


BOGfM 


inWK  • 
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The  ACEC  under  the 
Test  Harness 


Pretest 

Interactive  /  Manual 

Entering  results 

Automatic  /  Manual 

Performance  tests 

Groups 

Batch 

Individual  tests 

Interactive  /  Direct 

Entering  results 

Automatic 

Analysis 

Interactive  /  Batch 

j 

Workshop 

BOCINQ 

^nvn  9 

The  ACEC  under  the 

Test  Harness 

A 

Assessors 

Debugger 

Interactive  /  Manual 

Entering  results 

Manual 

Diagnostics 

Interactive  /  Manual 

Entering  results 

Manual 

Library 

Interactive  /  Manual  /  Batch 

Entering  results 

Manual  /  Automatic 

Capacity 

Batch 

Entering  results 

Automatic 

j 

WorfcStiep 

BOBNQ 

10 
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Performance  Tests: 
AES  /  ACEC  Map 


AES  group 

ACEC  group 

A 

compiler  performance  tests 

SY 

1 

general  run-time  efficiency  tests 

various 

J 

NPL  Performance  Test  Suite 

various 

K 

tasking  tests  for  MASCOT  systems 

TK 

L 

general  tasking  tests 

TK 

M 

storage  management  tests 

SR 

N 

input/output  tests 

10 

0 

optimization  tests 

OP 

R 

implementation  dependency  tests 

various 

^  V 

benchmark  tests 

CL  J 
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Analysis 


Menu 
Input  data 

Single  System  Analysis  -  AES  reports 
Comparative  Aniysis 


Wotk  Shop  BOEINQ 


AES  Assessors 


Similar  Assessors 
Candidates  for  inclusion 
Non-candidates 

V _ ) 

workshop  BOBNQ 


ACEC/AES  MERGER  WORKSHOP 


C-83 


Asessors:  AES  /  ACEC  Map 


AES  group 

- 1 

ACEC  group 

B..  F 

compiler  information  tests 

YD 

G 

compiler  capacity  tests 

YC 

Q 

run-time  limit  tests 

YC 

S 

erroneous  execution  tests 

YD 

T 

incorrect  order  dependency  tests  YD 

U 

iinker/loader  tests 

YL,SY 

D* 

debugger  tests 

YB 

LS 

PLS  scenario  support  tests 

YL 

source  generator  tests 

Debugger  Assessor: 
AES  versus  ACEC 

] 

AES 

ACEC 

Number  of  questions 

272 

118 

Test  programs 

11  + 

36 

Scenarios 

5 

29 

Detail  of  scenarios 

less 

more 

J 

Workshop 
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Debugger  Assessor: 

AES  Areas  with  Little  Overlap 


A 


AES 

ACEC 

Documentation 

25 

0 

Source  display 

15 

2 

Error  handling 

13 

0 

Macros 

9 

2 

Tracing 

17 

1 

Private  types 

7 

0 

Heap 

6 

0 

Debugging  file  iO 

3 

0 

Performance 

16 

4 

Capacity 

13 

1 

Overail  summary 

10 

0 

WoffcSlMp 


BOEINO 


inMi  1« 


Duplicate  Assessors 


Merge  selected  tests 


Review  AES  approach 


Review  for  easier  portability 


WerfcS^op 


BOCINO 


inl«2  20 
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Assessors:  AES  only 


IT 

E* 

N* 

P* 

R* 

T* 


V 


command  language  interpreter 
editor  tests 
name  expander  tests 
pretty  printer  tests 
requirements  analyses  tests 
test  support  toolset  tests 
test  bed  generator 
stub  generator 
test  coverage  analysis 
timing  analysis  (profiler) 
assertion  checker 

version  and  configuration  control  tests 
cross<reference  tests 


Work  Mop 


BOBNO 


insm  21 


Assessors: 

Candidates  for  inclusion 


V _ 

WpfkttMp 


Profiler 

Test  coverage  analysis 
Cross  reference 
Pretty  printing 
Syntax  based  editing 
Test  bed  generator 
Name  expander 
Stub  generator 
Assertion  checker 


BOBNO 


IflMB  22 
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Summary 

Interface 

Database 

Command  file  generator 
AES  Performance  Tests 
Review  and  merge 

Analysis 

AES  Assessors 

Review  and  merge 

Make  others  available 

J 

WomShep 

BOBNa 

intm  24 
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